Augmented reality navigation systems for use with robotic surgical systems and methods of their use

ABSTRACT

The present disclosure is directed to augmented reality navigation systems and methods of their use that, inter alia, address the need for systems and methods of robotic surgical system navigation with reduced distraction to surgeons. Augmented reality navigation systems disclosed herein enable a surgeon to maintain focus on a surgical site and/or surgical tool being used in a surgical procedure while obtaining a wide range of navigational information relevant to the procedure. Navigational information can appear in the augmented reality navigation system as being presented on virtual displays that sit in a natural field of view of a surgeon during a procedure. Navigational information can also appear to be overlaid over a patient&#39;s anatomy. Augmented reality navigation systems comprise a head mounted display comprising an at least partially transparent display screen, at least one detector connected to the head mounted display for identifying real-world features, and a computer subsystem.

FIELD

The present invention relates generally to augmented reality systems foruse with robotic surgical systems and methods of their use.

BACKGROUND

Robotic surgical systems are used in many surgical procedures in orderto assist surgeons in precisely and accurately performing theprocedures. Frequently, these procedures require precise placement ofone or more implants and can be performed using minimally invasivetechniques. Robotic surgical systems follow pre-planned orintra-operatively planned trajectories that assist the surgeon inplacing implants while maintaining their intended alignment. Navigationmarkers placed throughout the surgical environment are used to registerthe environment (e.g., patient anatomy) with the robotic surgical systemin order to properly orient the robot to the pre-planned orintra-operatively planned trajectories. Additionally, medical image datacan be registered to the robotic surgical system to provide a model ofthe patient's anatomy for use in navigation.

Surgeons plan and monitor trajectories as well as monitor status of arobotic surgical system and a patient's anatomy during a procedure usinga fixed display, for example, attached to or next to the roboticsurgical system. Such a fixed display is the primary mechanism fornavigating and monitoring a robotic surgical system during a procedure.This is especially true for minimally invasive procedures where apatient's anatomy obstructs direct view of the surgical site. However,fixed displays require a surgeon to divert his or her vision away fromthe surgical site and/or surgical tools that he or she is manipulatingin order to obtain navigational information displayed on the screen.Moreover, the display screen can physically obstruct a surgeon's view ofa portion of the surgical environment.

SUMMARY

There is a need for systems and methods for viewing navigationalinformation from a robotic surgical system that reduce a surgeon's needto divert his or her vision while not obstructing view of the surgicalenvironment. The present disclosure is directed to augmented realitynavigation systems and methods of their use that, inter alia, addressthe need for systems and methods of robotic surgical system navigationwith reduced distraction to surgeons. Augmented reality navigationsystems disclosed herein enable a surgeon to maintain focus on asurgical site and/or surgical tool being used in a surgical procedurewhile obtaining a wide range of navigational information relevant to theprocedure. Navigational information includes, but is not limited to, amodel of a patient's anatomy derived from medical image data, atrajectory or position of a surgical tool or robotic surgical system, ora position and orientation of a surgical implant. Navigationalinformation can be sent to an augmented reality navigation system asnavigation input data from a robotic surgical system. Navigationalinformation can appear in the augmented reality navigation system asbeing presented on virtual displays that sit in a natural field of viewof a surgeon during a procedure. Navigational information can alsoappear to be overlaid over a patient's anatomy. Navigational informationcan include information otherwise not visible in a surgeon's naturalfield of view, for example trajectories and or portions of a surgicaltool obscured by a patient's anatomy.

Augmented reality navigation systems comprise a head mounted displaycomprising an at least partially transparent display screen, at leastone detector connected to the head mounted display for identifyingreal-world features, and a computer subsystem. The display screendisplays augmentation graphics, for example navigation augmentationgraphics that provide navigational information to a surgeon. Thenavigation augmentation graphics can appear as a separate display in thefield of view of a surgeon or overlaid over a patient's anatomy. The atleast one detector identifies real-world features, wherein thereal-world features can be, for example fiducials and/or patient anatomyrecognized via image recognition methods. In this way, the at least onedetector mounted to the head mounted display acts as the detector in atypical navigation system used during surgery (e.g., can be used toregister a patient's anatomy and a robotic surgical system) withoutrequiring an additional piece of equipment in the surgical environment.The computer subsystem can be configured to perform a variety ofnavigational tasks useful to a surgeon during a procedure including, forexample, trajectory planning and execution. A motion sensor canoptionally be included to detect motion of the head of a surgeon wearingthe augmented reality navigation system providing additionalfunctionality and/or performance (e.g., a selection input means or driftcorrection).

In certain embodiments, an augmented reality navigation systemeliminates the need for an auxiliary navigation subsystem such as thosecommonly used with current robotic surgical systems. The at least onedetector in the augmented reality navigation system detects real-worldfeatures (e.g., fiducials) in sufficiently quantity and resolution as toproperly register a patient to a robotic surgical system and,optionally, one or more models of patient anatomy derived from medicalimage data. Therefore, the augmented reality navigation system acts as astandalone system without the need for additional equipment. Although,in certain embodiments, an auxiliary detector is used in conjunctionwith the augmented reality navigation system. An auxiliary detector mayprovide a larger registered field, improved resolution of registration,and/or redundancy.

In one aspect, the invention is directed to an augmented realitynavigation system for use with a robotic surgical system, the systemcomprising: a head mounted display comprising an at least partiallytransparent display screen configured to display augmentation graphics(e.g., semi-opaque images) (e.g., navigation augmentation graphics)which appear to a user to be superimposed on at least a portion of anatural field of view of the user; at least one detector for identifyingreal-world features, the at least one detector connected to the headmounted display [e.g., wherein the at least one detector comprises atleast one of an optical camera (e.g., a video camera), an EMF detector,a LiDAR detector, an acoustic detector, and an RF detector] [e.g.,wherein the real-world features comprises fiducials and/or identifiedpatient anatomy (e.g., wherein the real-world features are fiducialsconnected to at least one of a patient, a surgical tool, and the roboticsurgical system (e.g., a robotic arm, a part of a robotic arm, and/or anend-effector of a robotic arm))]; a processor of a computing device; anda non-transitory computer readable medium having instructions storedthereon, wherein the instructions, when executed by the processor, causethe processor to: receive, by the processor, a detector input signalfrom the at least one detector, wherein the detector input signalcorresponds to a field of view of the at least one detector and thefield of view comprises at least a portion of anatomy of a patientduring a surgical procedure, determine, by the processor, a relativelocation and/or orientation for each of one or more the real-worldfeatures in the detector input signal, generate and/or access, by theprocessor, a representation of at least a portion of a surgical tooland/or a trajectory of the surgical tool, wherein the surgical tool isinserted into or connected to the robotic surgical system (e.g., whereinthe portion of the surgical tool is hidden from the natural field ofview of the user, e.g., within a patient), modify (e.g., least one ofrotate, scale, and translate), by the processor, at least a portion ofthe representation based on the relative location and/or orientation ofthe one or more real-world features, thereby forming an updatedrepresentation, render, by the processor, surgical tool augmentationgraphics based on the updated representation, and display, by theprocessor, the surgical tool augmentation graphics on the display screen(e.g., display, via the at least partially transparent display screen ofthe head mounted display, the surgical tool augmentation graphicssuperimposed on at least a portion of the natural field of view of theuser).

In some embodiments, the instructions cause the processor to: render, bythe processor, a surgical tool augmentation graphic for each of aplurality of surgical tool trajectories (e.g., planned surgical tooltrajectories); and display, by the processor, on the display screen, theplurality of surgical tool augmentation graphics such that the surgicaltool augmentation graphics appear overlaid on the anatomy of the patientand each of the trajectory augmentation graphics indicate a physicaltrajectory that could be followed during the surgical procedure.

In some embodiments, the instructions cause the processor to: determine,by the processor, a relative location and/or orientation for each of atleast one real-world feature from the detected input signal; modify, bythe processor, (e.g., by at least one of rotation, scaling, andtranslation) an anatomical model of a patient (e.g., a 3D model) basedon the relative locations and/or orientations determined from thedetected input signal, thereby forming an updated anatomical model(e.g., that is registered to the anatomy of the patient); render, by theprocessor, anatomical model augmentation graphics based at least in parton the updated anatomical model; and display, by the processor, on thedisplay screen, the anatomical model augmentation graphics such that theupdated anatomical model appears overlaid on the anatomy of the patient.

In some embodiments, the augmented reality navigation system comprises amotion sensor (e.g., an inertial motion unit (IMU)) connected to thehead mounted display for outputting a motion signal based on measuredmotion of the head mounted display and wherein the instructions causethe processor to: update, by the processor, the relative position andorientation of the determined real-world features in the detector inputsignal based on motion detected by the motion sensor; and update, by theprocessor, the surgical tool augmentation graphics based on the updatedrelative position and orientation.

In some embodiments, the instructions cause the processor to: receive,by the processor, a user input trajectory selection signal that selectsa trajectory from a set of one or more planned trajectories (e.g., oneor more preoperatively or intraoperatively planned trajectories) (e.g.,wherein the user input trajectory selection signal corresponds to agesture or sound made by the user or a position and/or orientation of arobotic arm and/or end effector of the robotic surgical system);determine, by the processor, a selected trajectory based at least inpart on the user input trajectory selection signal; and automaticallymove, by the processor, a robotic arm and/or end effector of the roboticsurgical system to be aligned with the selected trajectory.

In some embodiments, the instructions cause the processor to:automatically move, by the processor, the robotic arm and/or endeffector of the robotic surgical system along the selected trajectory(e.g., towards the anatomy of the patient).

In some embodiments, the instructions cause the processor to: defineand/or update, by the processor, a haptic object that comprises theselected trajectory; and constrain, by the processor, motion of arobotic arm and/or end effector such that motion of at least a portionof the surgical tool inserted into or attached to the robotic arm and/orend effector is constrained to within the haptic object.

In some embodiments, the at least one detector comprises a detector withat least a minimum field of view of 40 degrees (e.g., as measured on adiagonal). In some embodiments, the display screen has a resolution ofat least 1280×720 pixels.

In some embodiments, the augmented reality navigation system comprises apointer tool for making surgical planning selections (e.g., of atrajectory and/or position(s) and/or orientation(s) that define atrajectory), wherein the pointer tool is configured to be detected bythe at least one detector.

In some embodiments, the instructions cause the processor to registeranatomy of a patient with the robotic surgical system, the augmentedreality navigation system, and, optionally, an anatomical model of thepatient based on medical image data (e.g., X-ray data, CT data, MRIdata, fluoroscopy data).

In some embodiments, the at least one detector comprises a video cameraand the instructions cause the processor to: generate, by the processor,a video signal based on the detector input signal; and output, by theprocessor, the video signal for display on at least one of (i) a monitorand (ii) a second head mounted display comprising an at least partiallytransparent display screen configured to display augmentation graphics(e.g., semi-opaque images) which appear to a user to be superimposed onat least a portion of a natural field of view of the user.

In some embodiments, the system comprises one or more fiducial markersconnected to the head mounted display. In some embodiments, theinstructions cause the processor to: receive, by the processor, arelative location and orientation of the one or more fiducial markersconnected to the head mounted display, wherein the one or more fiducialmarkers are detected by a secondary detector (e.g., not physicallyconnected to the head mounted display) (e.g., an EMF detector, an RFdetector, an acoustic detector, a LiDAR detector, an optical detector);and modify (e.g., at least one of rotate, scale, and translate) at leastone of (i) an anatomical model, (ii) a representation of a surgicalimplant, (iii) a representation of a trajectory of a surgical tool, and(iv) a representation of at least a portion of a surgical tool hiddenfrom a natural field of view based on the one or more fiducial markersdetected by the secondary detector.

In some embodiments, the instructions cause the processor to: receive,by the processor, a relative location and orientation of one or morereal-world features detected by a secondary detector (e.g., notphysically connected to the head mounted display) (e.g., an EMFdetector, an RF detector, an acoustic detector, a LiDAR detector, anoptical detector); modify (e.g., at least one of rotate, scale, andtranslate), by the processor, at least one of (i) an anatomical model,(ii) a representation of a surgical implant, (iii) a representation of atrajectory of a surgical tool, and (iv) a representation of at least aportion of a surgical tool hidden from a natural field of view based onthe one or more real-world features detected by the secondary detector;render and/or update, by the processor, updated augmentation graphicsbased at least in part on the modified at least one of (i), (ii), (iii),and (iv); an display, by the processor, on the display screen, theupdated augmentation graphics.

In some embodiments, the surgical procedure comprises at least one of aspinal surgical procedure, an orthopedic surgical procedure, anorthopedic trauma surgical procedure, and a neurosurgical procedure. Insome embodiments, the surgical procedure comprises a minimally invasivesurgical procedure.

In one aspect, the invention is directed to an augmented realitynavigation system for use with a robotic surgical system, the systemcomprising: a head mounted display comprising an at least partiallytransparent display screen configured to display augmentation graphics(e.g., semi-opaque images) (e.g., navigation augmentation graphics)which appear to a user to be superimposed on at least a portion of anatural field of view of the user; at least one detector for identifyingreal-world features, the at least one detector connected to the headmounted display [e.g., wherein the at least one detector comprises atleast one of an optical camera (e.g., a video camera), an EMF detector,a LiDAR detector, an acoustic detector, and an RF detector] [e.g.,wherein the real-world features comprises fiducials and/or identifiedpatient anatomy (e.g., wherein the real-world features are fiducialsconnected to at least one of a patient, a surgical tool, and the roboticsurgical system (e.g., a robotic arm, a part of a robotic arm, and/or anend-effector of a robotic arm))]; and a computer subsystem configured togenerate and/or access a representation of at least a portion of asurgical tool and/or a trajectory of the surgical tool during a surgicalprocedure, modify at least a portion of the representation based on arelative position and/or orientation of one or more real-world featuresin a detector input signal received from the at least one detector, anddisplay, on the display screen, surgical tool augmentation graphicsbased on the modified representation, wherein the surgical tool isinserted into or connected to the robotic surgical system (e.g., whereinthe portion of the surgical tool is hidden from the natural field ofview of the user, e.g., within a patient).

In some embodiments, the computer subsystem is configured to render asurgical tool augmentation graphic for each of a plurality of surgicaltool trajectories (e.g., planned surgical tool trajectories), anddisplay, on the display screen, the plurality of surgical toolaugmentation graphics such that the surgical tool augmentation graphicsappear overlaid on the anatomy of the patient and each of the trajectoryaugmentation graphics indicate a physical trajectory that could befollowed during the surgical procedure.

In some embodiments, the computer subsystem is configured to modify(e.g., by at least one of rotation, scaling, and translation) ananatomical model of a patient (e.g., a 3D model) based on one or morerelative location(s) and/or orientation(s) determined from the detectedinput signal, thereby forming an updated anatomical model (e.g., that isregistered to the anatomy of the patient), and the computer subsystem isconfigured to display, on the display screen, anatomical modelaugmentation graphics corresponding to the updated anatomical model suchthat the updated anatomical model appears overlaid on the anatomy of thepatient.

In some embodiments, the augmented reality navigation system comprises amotion sensor (e.g., an inertial motion unit (IMU)) connected to thehead mounted display for outputting a motion signal based on measuredmotion of the head mounted display, wherein the computer subsystem isconfigured to update the surgical tool augmentation graphics based onmotion detected by the motion sensor.

In some embodiments, the computer subsystem is configured to determine aselected trajectory based at least in part on a user input trajectoryselection signal that selects the selected trajectory from a set of oneor more planned trajectories (e.g., one or more preoperatively orintraoperatively planned trajectories) (e.g., wherein the user inputtrajectory selection signal corresponds to a gesture or sound made bythe user or a position and/or orientation of a robotic arm and/or endeffector of the robotic surgical system), and automatically move arobotic arm and/or end effector of the robotic surgical system to bealigned with the selected trajectory.

In some embodiments, the computer subsystem is configured toautomatically move the robotic arm and/or end effector of the roboticsurgical system along the trajectory (e.g., towards the anatomy of thepatient).

In some embodiments, the computer subsystem is configured to define ahaptic object that comprises the trajectory and constrain motion of arobotic arm and/or end effector such that motion of at least a portionof a surgical tool attached to the robotic arm and/or end effector isconstrained to within the haptic object.

In some embodiments, the at least one detector comprises a detector withat least a minimum field of view of 40 degrees (e.g., as measured on adiagonal). In some embodiments, the display screen has a resolution ofat least 1280×720 pixels. In some embodiments, the augmented realitynavigation system comprises a pointer tool for making surgical planningselections, wherein the pointer tool is configured to be detected by theat least one detector.

In some embodiments, the computer subsystem is configured to registeranatomy of a patient with the robotic surgical system, the augmentedreality navigation system, and, optionally, an anatomical model of thepatient based on medical image data (e.g., X-ray data, CT data, MRIdata, fluoroscopy data).

In some embodiments, the computer subsystem is configured to generate avideo signal based on the detector input signal and output the videosignal for display on at least one of (i) a monitor and (ii) a secondhead mounted display comprising an at least partially transparentdisplay screen configured to display augmentation graphics (e.g.,semi-opaque images) which appear to a user to be superimposed on atleast a portion of a natural field of view of the user.

In some embodiments, the system comprises one or more fiducial markersconnected to the head mounted display. In some embodiments, the computersubsystem is configured to receive a relative location and orientationof the one or more fiducial markers connected to the head mounteddisplay detected by a secondary detector (e.g., not physically connectedto the head mounted display) (e.g., an EMF detector, an RF detector, anacoustic detector, a LiDAR detector, an optical detector) and modify(e.g., at least one of rotate, scale, and translate) at least one of (i)an anatomical model, (ii) a representation of a surgical implant, (iii)a representation of a trajectory of a surgical tool, and (iv) arepresentation of at least a portion of a surgical tool hidden from anatural field of view based on the one or more fiducial markers detectedby the secondary detector.

In some embodiments, the computer subsystem is configured to receive arelative location and orientation of one or more real-world featuresdetected by a secondary detector (e.g., not physically connected to thehead mounted display) (e.g., an EMF detector, an RF detector, anacoustic detector, a LiDAR detector, an optical detector) and modify(e.g., at least one of rotate, scale, and translate) at least one of (i)an anatomical model, (ii) a representation of a surgical implant, (iii)a representation of the trajectory, and (iv) a representation of atleast a portion of a surgical tool hidden from a natural field of viewbased on the one or more real-world features detected by the secondarydetector, and the computer subsystem is configured to display, on thedisplay screen, updated augmentation graphics based at least in part onthe modified at least one of (i), (ii), (iii), and (iv).

In some embodiments, the surgical procedure comprises at least one of aspinal surgical procedure, an orthopedic surgical procedure, anorthopedic trauma surgical procedure, and a neurosurgical procedure. Insome embodiments, the surgical procedure comprises a minimally invasivesurgical procedure.

In one aspect, the invention is directed to a method of using anaugmented reality navigation system with a robotic surgical system, themethod comprising: providing and/or accessing the augmented realitynavigation system, wherein the augmented reality navigation systemcomprises: a head mounted display comprising an at least partiallytransparent display screen configured to display augmentation graphics(e.g., semi-opaque images) (e.g., navigation augmentation graphics)which appear to a user to be superimposed on at least a portion of anatural field of view of the user; optionally, a motion sensor (e.g., aninertial motion unit (IMU)) connected to the head mounted display foroutputting a motion signal based on measured motion of the head mounteddisplay; and at least one detector for identifying real-world features,the at least one detector connected to the head mounted display [e.g.,wherein the at least one detector comprises at least one of an opticalcamera (e.g., a video camera), an EMF detector, a LiDAR detector, anacoustic detector, and an RF detector] [e.g., wherein the real-worldfeatures comprises fiducials and/or identified patient anatomy (e.g.,wherein the real-world features are fiducials connected to at least oneof a patient, a surgical tool, and the robotic surgical system (e.g., arobotic arm, a part of a robotic arm, and/or an end-effector of arobotic arm))]; receiving (e.g., by a processor of a computer subsystem)a detector input signal from the at least one detector, wherein thedetector input signal corresponds to a field of view of the at least onedetector and the field of view comprises at least a portion of anatomyof a patient during a surgical procedure, determining (e.g., by aprocessor of a computer subsystem) a relative location and/ororientation for each of one or more the real-world features in thedetector input signal, generating and/or accessing (e.g., by a processorof a computer subsystem) a representation of at least a portion of asurgical tool and/or a trajectory of the surgical tool, wherein thesurgical tool is inserted into or connected to the robotic surgicalsystem (e.g., wherein the portion of the surgical tool is hidden fromthe natural field of view of the user, e.g., within a patient),modifying (e.g., least one of rotating, scaling, and translating) (e.g.,by a processor of a computer subsystem) at least a portion of therepresentation based on the relative location and orientation of the oneor more real-world features, thereby forming an updated representation,rendering (e.g., by a processor of a computer subsystem) surgical toolaugmentation graphics based on the updated representation, anddisplaying (e.g., by a processor of a computer subsystem) the surgicaltool augmentation graphics on the display screen (e.g., displaying, viathe at least partially transparent display screen of the head mounteddisplay, the surgical tool augmentation graphics superimposed on atleast a portion of the natural field of view of the user).

In some embodiments, the method comprises: rendering (e.g., by aprocessor of a computer subsystem) a surgical tool augmentation graphicfor each of a plurality of surgical tool trajectories (e.g., plannedsurgical tool trajectories); and displaying (e.g., by a processor of acomputer subsystem) on the display screen, the plurality of surgicaltool augmentation graphics such that the surgical tool augmentationgraphics appear overlaid on the anatomy of the patient and each of thetrajectory augmentation graphics indicate a physical trajectory thatcould be followed during the surgical procedure.

In some embodiments, the method comprises: determining (e.g., by aprocessor of a computer subsystem) a relative location and/ororientation for each of at least one real-world feature from thedetected input signal; modifying (e.g., by at least one of rotating,scaling, and translating) (e.g., by a processor of a computer subsystem)an anatomical model of a patient (e.g., a 3D model) based on therelative locations and/or orientations determined from the detectedinput signal, thereby forming an updated anatomical model (e.g., that isregistered to the anatomy of the patient); rendering (e.g., by aprocessor of a computer subsystem) anatomical model augmentationgraphics based at least in part on the updated anatomical model; anddisplaying, on the display screen, (e.g., by a processor of a computersubsystem) the anatomical model augmentation graphics such that theupdated anatomical model appears overlaid on the anatomy of the patient.

In some embodiments, the method comprises: updating (e.g., by aprocessor of a computer subsystem) the relative position and orientationof the determined real-world features in the detector input signal basedon motion detected by the motion sensor; and updating (e.g., by aprocessor of a computer subsystem) the surgical tool augmentationgraphics based on the updated relative position and orientation.

In some embodiments, the method comprises: receiving (e.g., by aprocessor of a computer subsystem) a user input trajectory selectionsignal that selects a trajectory from a set of one or more plannedtrajectories (e.g., one or more preoperatively or intraoperativelyplanned trajectories) (e.g., wherein the user input trajectory selectionsignal corresponds to a gesture or sound made by the user or a positionand/or orientation of a robotic arm and/or end effector of the roboticsurgical system); determining (e.g., by a processor of a computersubsystem) a selected trajectory based at least in part on the userinput trajectory selection signal; and automatically (e.g., by aprocessor of a computer subsystem) moving a robotic arm and/or endeffector of the robotic surgical system to be aligned with the selectedtrajectory.

In some embodiments, the method comprises: automatically (e.g., by aprocessor of a computer subsystem) moving the robotic arm and/or endeffector of the robotic surgical system along the selected trajectory(e.g., towards the anatomy of the patient). In some embodiments, themethod comprises: defining and/or updating (e.g., by a processor of acomputer subsystem) a haptic object that comprises the selectedtrajectory; and constraining motion of a robotic arm and/or end effectorsuch that motion of at least a portion of the surgical tool insertedinto or attached to the robotic arm and/or end effector is constrainedto within the haptic object.

In some embodiments, the at least one detector comprises a detector withat least a minimum field of view of 40 degrees (e.g., as measured on adiagonal). In some embodiments, the display screen has a resolution ofat least 1280×720 pixels.

In some embodiments, the method comprises: registering anatomy of apatient with the robotic surgical system, the augmented realitynavigation system, and, optionally, an anatomical model of the patientbased on medical image data (e.g., X-ray data, CT data, MRI data,fluoroscopy data).

In some embodiments, the at least one detector comprises a video cameraand the method comprises: generating (e.g., by a processor of a computersubsystem) a video signal based on the detector input signal; andoutputting (e.g., by a processor of a computer subsystem) the videosignal for display on at least one of (i) a monitor and (ii) a secondhead mounted display comprising an at least partially transparentdisplay screen configured to display augmentation graphics (e.g.,semi-opaque images) which appear to a user to be superimposed on atleast a portion of a natural field of view of the user.

In some embodiments, the system comprises one or more fiducial markersconnected to the head mounted display and the method comprises:receiving (e.g., by a processor of a computer subsystem) a relativelocation and orientation of the one or more fiducial markers connectedto the head mounted display, wherein the one or more fiducial markersare detected by a secondary detector (e.g., not physically connected tothe head mounted display) (e.g., an EMF detector, an RF detector, anacoustic detector, a LiDAR detector, an optical detector); and modifying(e.g., at least one of rotating, scaling, and translating) (e.g., by aprocessor of a computer subsystem) at least one of (i) an anatomicalmodel, (ii) a representation of a surgical implant, (iii) arepresentation of a trajectory of a surgical tool, and (iv) arepresentation of at least a portion of a surgical tool hidden from anatural field of view based on the one or more fiducial markers detectedby the secondary detector.

In some embodiments, the method comprises: receiving (e.g., by aprocessor of a computer subsystem) a relative location and orientationof one or more real-world features detected by a secondary detector(e.g., not physically connected to the head mounted display) (e.g., anEMF detector, an RF detector, an acoustic detector, a LiDAR detector, anoptical detector); modifying (e.g., at least one of rotating, scaling,and translating) (e.g., by a processor of a computer subsystem) at leastone of (i) an anatomical model, (ii) a representation of a surgicalimplant, (iii) a representation of a trajectory of a surgical tool, and(iv) a representation of at least a portion of a surgical tool hiddenfrom a natural field of view based on the one or more real-worldfeatures detected by the secondary detector; rendering and/or updating(e.g., by a processor of a computer subsystem) updated augmentationgraphics based at least in part on the modified at least one of (i),(ii), (iii), and (iv); and displaying (e.g., by a processor of acomputer subsystem) on the display screen, the updated augmentationgraphics.

In some embodiments, the surgical procedure comprises at least one of aspinal surgical procedure, an orthopedic surgical procedure, anorthopedic trauma surgical procedure, and a neurosurgical procedure. Insome embodiments, the surgical procedure comprises a minimally invasivesurgical procedure.

Definitions

In order for the present disclosure to be more readily understood,certain terms used herein are defined below. Additional definitions forthe following terms and other terms may be set forth throughout thespecification.

In this application, the use of “or” means “and/or” unless statedotherwise. As used in this application, the term “comprise” andvariations of the term, such as “comprising” and “comprises,” are notintended to exclude other additives, components, integers or steps. Asused in this application, the terms “about” and “approximately” are usedas equivalents. Any numerals used in this application with or withoutabout/approximately are meant to cover any normal fluctuationsappreciated by one of ordinary skill in the relevant art. In certainembodiments, the term “approximately” or “about” refers to a range ofvalues that fall within 25%, 20%, 19%, 18%, 17%, 16%, 15%, 14%, 13%,12%, 11%, 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, or less in eitherdirection (greater than or less than) of the stated reference valueunless otherwise stated or otherwise evident from the context (exceptwhere such number would exceed 100% of a possible value).

The following description generally makes use of a Cartesian coordinatesystem in describing positions, orientations, and directions of travelof various elements of and relating to the systems and methods describedherein. However, it should be understood that specific coordinates(e.g., “x, y, z”) and related conventions based on them (e.g., a“positive x-direction”, an “x, y, or z-axis”, an “xy, xz, or yz-plane”,and the like) are presented for convenience and clarity, and that, asunderstood by one of skill in the art, other coordinate systems could beused (e.g., cylindrical, spherical) and are considered to be within thescope of the claims.

Navigational information: As used herein, the term “navigationalinformation” means information useful in navigating during a surgicalprocedure. In certain embodiments, navigating includes navigating one ormore surgical tools and/or implants (or other surgical apparatus). Thesurgical tool(s) may be attached to a robotic surgical system.Navigational information includes, but is not limited to, one or more ofsurgical trajectories, positions and/or orientations of (i) surgicaltools and/or apparatus (e.g., implants) and/or surgical equipment (e.g.,surgical tables), patient anatomy and/or models thereof, medical imagedata, and positions and/or orientations of a robotic surgical system. Asused herein, where an augmented reality navigation system is describedas displaying navigational information to a surgeon, it is understoodthat other information not immediately relevant to navigation, butrelevant generally to a surgical procedure may also be displayed (e.g.,in a similar fashion). For example, patient health information regardinga patient's vitals or condition (e.g., patient history) or statusinformation related to a surgical procedure (e.g., progress,instructions, or other information) may be displayed (e.g., on a virtualdisplay presented on a display screen of an augmented reality navigationsystem). When appropriate, navigational information can optionallyappear overlaid over a patient's anatomy.

Augmentation graphic: As used herein, the term “augmentation graphic”refers to a graphic that is rendered by a processor and displayed on adisplay screen of an augmented reality navigation system such that thegraphic appears superimposed on the natural field of view of a surgeonas viewed through the display screen. In certain embodiments, anaugmentation graphic may also be rendered for display on a remotemonitor to allow third party observers to observe what a surgeon isseeing (e.g., one that is mounted on a wall of an operating room). Anaugmentation graphic may be a standalone graphic that appears in thenatural field of view (e.g., as a virtual display floating in the fieldof view of a surgeon). An augmentation graphic may appear overlaid overone or more objects (e.g., patient anatomy) in the natural field ofview, such that the augmentation graphic coincides with a physicalobject (e.g., portion of a patient anatomy) that is represented by theaugmentation graphic (e.g., wherein the physical object or a portionthereof is not otherwise seen by a surgeon). In certain embodiments, anaugmentation graphic may appear as a 3D object that sits adjacent to thephysical object that it represents (e.g., with a common orientation butoffset by a spatial translation). In some embodiments, augmentationgraphics comprise several graphics for display in a chronological ordersuch that they appear as a video on a display screen that augments asurgeon's natural field of view. For example, a surgeon can view aportion of a procedure as it will occur overlaid over a patient'sphysical anatomy.

Medical image data: As used herein, medical image data refers to imagedata that represents at least a portion of a patient. Medical image datamay be generated using any suitable technique including, but not limitedto, X-ray techniques, radiography techniques (fluoroscopy techniques(e.g., generated using an O-arm or C-arm)), tomographic techniques(e.g., X-ray computed tomography, positron emission tomography (PET), ormagnetic resonance imaging (MRI)), ultrasound techniques, orelastography techniques.

Pointer tool: As used herein, the term “pointer tool” refers to a toolthat is used to indicate a desired position and/or orientation. Apointer tool may be configured to be inserted into a robotic surgicalsystem, for example, an end effector thereof or a tool guide attachedthereto. A pointer tool may be a specially configured instrument solelyused for pointing. A surgical tool may be used as a pointer tool. Forexample, a drill bit, a drill guide, a tool guide, an awl, or similarsurgical tool may be used as a pointer tool. A pointer tool may have oneor more fiducials attached to the tool (e.g., sufficient to determine aposition and orientation of the pointer tool (e.g., by triangulation)).

Real-world feature: As used herein, the term “real-world feature” refersto a physical object or portion thereof that can be detected by adetector such that spatial information about the object can bedetermined. Spatial information comprises a position and/or orientation.A real-world feature may be identified patient anatomy (e.g., identifiedby reference to an anatomy database (e.g., a database of images ofpatient anatomy)) that is detected using one or more image-recognitiontechniques. A real-world feature may be any suitable fiducial. Afiducial may be attached to, for example, surgical equipment (e.g., anoperating table), a patient, a surgical tool, an implant, a roboticsurgical system, or an augmented reality navigation system (e.g., on thehead mounted display). A fiducial may comprise a plurality of markers toassist in orienting the fiducial in the environment during navigation(e.g., tracking). For example, in certain embodiments, a fiducialcomprises a plurality of spatially separated markers (e.g., 3 markers or4 markers) attached to a rigid holding apparatus that is attachable toan object, wherein each of the markers is configured to be detected by adetector disposed on the head mounted display (e.g., emit or alter anelectromagnetic field for an EMF detector or have a certain reflectivityfor an optical detector). Real-world features are used to determineposition and orientation of objects in a surgical environment. Anexample of a technique used to make such a determination using thesystems disclosed herein is triangulation, however, it will be apparentto those of ordinary skill in the art of patient registration that anyof a number of techniques may be used, alone or in combination, such assurface matching or other similar correlation techniques.

Trajectory: As used herein, the term “trajectory” refers to a pathdesired and/or intended to be followed. A trajectory may be modeled andgraphics representing the trajectory may be displayed on a displayscreen of an augmented reality navigation system. A trajectory may alinear trajectory (e.g., wherein all points along the trajectory fallonto a line) or a non-linear trajectory. A trajectory sent by aprocessor (e.g., to a robotic surgical system) or stored on a computerreadable medium (e.g., for manipulation by a processor) may berepresented by any data sufficient to define the trajectory.Non-limiting examples of data used to define a trajectory include asingle coordinate in space and an orientation (e.g., vector), aplurality of coordinates, and a functional relationship (e.g., involvingan x, y, and z variable). In certain embodiments, a path may be followedusing a robotic surgical system (e.g., automatically) or manually by asurgeon.

Robotic surgical system: As used herein, the term “robotic surgicalsystem” refers to a system comprising a robotic arm configured to assistin a surgical procedure. A robotic arm may assist in a surgicalprocedure by holding and/or manipulating (e.g., guiding and/or moving)one or more surgical tool(s). In certain embodiments, a robotic surgicalsystem comprises an active, non-backdriveable robotic arm. In certainembodiments, a robotic surgical system comprises a passive,backdriveable robotic arm. In certain embodiments, a robotic surgicalsystem is configured to be manipulated directly by a surgeon (e.g., bygrasping and maneuvering). In certain embodiments, a robot is configuredto be manipulated remotely by a surgeon (e.g., similarly to amaster/slave system). In certain embodiments, a robotic arm of a roboticsurgical system is configured to assist in a surgical procedure bypreventing a surgeon from actively maneuvering a surgical tool attachedto the robotic arm outside of a defined haptic object (i.e., hapticvolume). In certain embodiments, a robotic arm of a robotic surgicalsystem is configured to automatically move, upon input from a surgeon,onto and/or along a trajectory, such as a trajectory pre-operatively orintra-operatively planned using an augmented reality navigation systemin communication with the robotic surgical system. In certainembodiments, a robotic surgical system and an augmented realitynavigation system have a common computer subsystem.

BRIEF DESCRIPTION OF THE DRAWING

Drawings are presented herein for illustration purposes, not forlimitation. The foregoing and other objects, aspects, features, andadvantages of the invention will become more apparent and may be betterunderstood by referring to the following description taken inconjunction with the accompanying drawings, in which:

FIG. 1 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 2 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 3 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 4 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 5 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 6 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 7 illustrates a head mounted display apparatus that can be worn ona user's head and operates in accordance with some embodiments of theinvention;

FIG. 8 illustrates operations and methods that may be performed by anaugmented reality navigation system that includes a head mounted displayto display virtual display panels through a display screen of the headmounted display, according to some illustrative embodiments of theinvention;

FIG. 9 is a block diagram of electronic components of a computersubsystem coupled to head mounted display in an augmented realitynavigation system, according to illustrative embodiments of theinvention;

FIG. 10 is a block diagram of components of an augmented realitynavigation system that tracks the location of equipment (surgical tools,a surgeon's head mounted display, and parts of a patient's anatomy, andgenerates a three dimensional (3D) model from patient data that isdisplayed on the head mounted display to be rendered super-imposed at avisually aligned location on the patient's body in accordance withillustrative embodiments of the invention;

FIG. 11 is another block diagram of the electronic components andmodules of an augmented reality surgical system, in accordance withillustrative embodiments of the invention;

FIG. 12 is a block diagram of an exemplary method for using an augmentedreality navigation system to display a graphical representation of asurgical tool and/or its trajectory, according to illustrativeembodiments of the invention;

FIG. 13 is a block diagram of an exemplary method for using an augmentedreality navigation system with fixed crosshairs, according toillustrative embodiments of the invention;

FIG. 14 is a block diagram of an exemplary method for using an augmentedreality navigation system with a pointer tool, according to illustrativeembodiments of the invention;

FIG. 15 illustrates navigation information that may be displayed on anaugmented reality navigation system during a surgical procedure,according to illustrative embodiments of the invention;

FIG. 16 schematically illustrates portions of a model of a patientanatomy and an implant being inserted along a pre-planned orintra-operatively planned trajectory using a surgical tool connected toa robotic surgical system that are otherwise not visible to the wearer,as displayed on an augmented reality navigation system, according toillustrative embodiments of the invention;

FIG. 17 is a block diagram of an example network environment for use inthe methods and systems described herein, according to illustrativeembodiments of the invention; and

FIG. 18 is a block diagram of an example computing device and an examplemobile computing device, for use in illustrative embodiments of theinvention.

DETAILED DESCRIPTION

It is contemplated that systems, devices, methods, and processes of theclaimed invention encompass variations and adaptations developed usinginformation from the embodiments described herein. Adaptation and/ormodification of the systems, devices, methods, and processes describedherein may be performed by those of ordinary skill in the relevant art.

Throughout the description, where articles, devices, and systems aredescribed as having, including, or comprising specific components, orwhere processes and methods are described as having, including, orcomprising specific steps, it is contemplated that, additionally, thereare articles, devices, and systems of the present invention that consistessentially of, or consist of, the recited components, and that thereare processes and methods according to the present invention thatconsist essentially of, or consist of, the recited processing steps.

It should be understood that the order of steps or order for performingcertain action is immaterial so long as the invention remains operable.Moreover, two or more steps or actions may be conducted simultaneously.

The mention herein of any publication, for example, in the Backgroundsection, is not an admission that the publication serves as prior artwith respect to any of the claims presented herein. The Backgroundsection is presented for purposes of clarity and is not meant as adescription of prior art with respect to any claim. Headers are providedfor the convenience of the reader and are not intended to be limitingwith respect to the claimed subject matter.

Augmented Reality Navigation Systems and Components Thereof

The augmented reality navigation systems disclosed herein comprise ahead mounted display comprising an at least partially transparentdisplay screen, at least one detector connected to (e.g., disposed on)the head mounted display for identifying real-world features, and acomputer subsystem. The computer subsystem can be configured to performa variety of navigational tasks useful to a surgeon during a procedureincluding, for example, trajectory planning and execution. A motionsensor can optionally be included to detect motion of the head of asurgeon wearing the augmented reality navigation system providingadditional functionality and/or performance (e.g., a selection inputmeans or drift correction). Augmented reality navigation systems for usein certain surgical procedures that are in accordance with certainembodiments of the present disclosure are described in U.S. PatentPublication No. 2016-0225192 A1, published on Aug. 4, 2016, thedisclosure of which is hereby incorporated by reference herein in itsentirety.

Embodiments of the present disclosure are directed to an augmentedreality surgical system that includes a head mounted display (HMD)apparatus that can be worn by a surgeon, physician, or other personnelduring a medical procedure. Throughout the present disclosure, where anaugmented reality navigation system is described as being worn by orused by a wearer, user, or surgeon, it is understood that any personassisting with, leading, or observing a surgical procedure canequivalently interact with an augmented reality navigation system in thesame manner as is being described. The user is not limited to any oneparticular individual with a specific relationship to a surgicalprocedure. The HMD can be configured to provide localized, real-timesituational awareness to the wearer. The HMD includes a display screenthat can be positioned within the natural line-of-sight and/or peripheryfield of view (FOV) of the wearer to provide visual information that canbe organized and displayed as a single virtual display or as acollection of virtual displays that a wearer can navigate between toview using head movement, hand gestures, voice commands, eye control,and/or other operations disclosed herein. In certain embodiments, adisplay screen can be manipulated to be in or out of a natural field ofview of a surgeon (e.g., by flipping the screen from an “in use” to “outof use” position”).

In certain embodiments, a surgeon or other person can wear an HMD to seea graphical representation of what is within the patient's body butcovered from view by skin, muscle, organs, skeletal structure, etc. Incertain embodiments, using an HMD can enable a surgeon to minimize thesize of an incision by observing where the incision needs to be made toreveal a targeted portion of the body. Similarly, an HMD can be usedwhen replacing a bone with prosthesis to enable a surgeon to observe areal-world feature that aids with orienting and moving surgical tool(s)(e.g., attached to a robotic surgical system) and/or a prosthesis duringthe procedure. An HMD may operate to improve the efficiency,productivity, throughput, and/or accuracy, and/or safety of the wearer'sperformance of a medical procedure. Moreover, an HMD can reduce mentalfatigue by reducing or eliminating a need for the wearer to referenceremote display devices having substantial angular offsets during amedical procedure.

In certain embodiments, an augmented reality navigation system has aninitialization time of under 10 minutes (e.g., under 5 minutes) therebyminimizing delay and interference of surgical procedures. In certainembodiments, an augmented reality navigation system can be comfortablyword for a minimum of one hour and up to four hours or more. In certainembodiments, an augmented reality navigation system is compatible withsurgeon's loupes. In certain embodiments, an augmented realitynavigation system is compatible with personal prescription glasses. Incertain embodiments, an augmented reality navigation system provides aminimum field of view of 40 degrees (e.g., 50 degrees, 60 degrees, 70degrees, 90 degrees, or more) as measured on a diagonal. In certainembodiments, an augmented reality navigation system comprises a headmounted display and computer subsystem coupled together to provide lessthan 50 millisecond latency (e.g., less than 20 millisecond latency)from the time that a display screen moves until updated navigationinformation is received by the display screen. In certain embodiments, adisplay screen can have a frame rate of 30 frames per second or more(e.g., 60 frames per second or 120 frames per second).

The augmented reality navigation systems disclosed herein comprise ahead mounted display comprising an at least partially transparentdisplay screen. The head mounted display comprising an at leastpartially transparent display screen can have a form factor similar toeyeglasses. For example, it can be binocular or monocular (i.e., thedisplay screen can be used to augment vision in one or both eyes). Incertain embodiments, the head mounted display is a binocular headmounted display. A binocular arrangement may be made of one contiguousdisplay screen at least partially covering both eyes when worn or it maybe two separate display screens (i.e., one screen for each eye). Thehead mounted display may be held on a wearer's head using armatures (asin a typical pair of eyeglasses) or some other mounting means (e.g., astrap, band, or fixture sized and shaped to be worn on a human head).The at least partially transparent display screen configured to displayaugmentation graphics can be any suitable commercially available displayscreen, for example, those used in heads up displays for piloting orathletics. The display screen is at least partially transparent suchthat a wearer can see at least a portion of their natural field of viewthrough the display screen while also seeing any augmentation graphicsdisplayed on the screen. In certain embodiments, the display screen is ahigh resolution display screen (e.g., has a resolution of at least1280×720).

The augmented reality navigation subsystems disclosed herein comprise atleast one detector. A detector is one suitable for use in determiningspatial information about real-world features in the field of view ofthe detector(s). A detector can be an optical camera (e.g., a videocamera), an EMF detector, a LiDAR detector, an acoustic detector, an RFdetector, or similar energy sensor. The detector receives informationfrom the environment in the field of view of the detector and produces adetector input signal that is sent to the computing subsystem. Incertain embodiments, a detector is disposed (e.g., attached) to a headmounted display). In certain embodiments, a detector is connected (e.g.,electrically) but spatially remote from the head mounted display. Incertain embodiments, fiducials of an appropriate type are selected andused during a surgical procedure based on the type of detector connectedto the head mounted display. For example, fiducials that output an EMFsignal are detected by an EMF detector mounted to the head mounteddisplay. One or more fiducials may be disposed on a head mounted displayfor tracking and/or orienting the head mounted display in the surgicalenvironment (e.g., as detected by an auxiliary detector or a secondaugmented reality navigation system). In certain embodiments, thedetector is an optical camera and image recognition is performed by thecomputer subsystem on the detector input signal in order to determineone or more real-world features. Detectors of different types can beused in combination in a single augmented reality navigation system. Twodetectors of the same type may be disposed on a head mounted display andspatially separated thereon (e.g., in order to triangulate real-worldfeatures). In certain embodiments, a detector is removable such that itcan be replaced by a detector of a different type. In certainembodiments, a detector coupled with a computer subsystem enablespatient registration accurate to within 2 mm (e.g., within 1.5 mm orwithin 1 mm) RMS for each position degree of freedom. In certainembodiments, one or more detector(s) (e.g., disposed on a head mounteddisplay) are configured to provide accurate registration throughout aperiod of time (e.g., a surgical procedure) over a certain spatialvolume in a surgical environment (e.g., a volume larger than a sphericalvolume of 2 foot radius, 4 foot radius, 6 foot radius, or larger). Incertain embodiments, such a volume may further be defined by a conewhose apex is located at the head mounted display.

In certain embodiments, an augmented reality navigation system comprisesa motion sensor. The motion sensor can track motion of a wearer's headover time. Such tracking can be used to, for example, smooth, or evenremove, jitter of augmentation graphics generated by natural smallmovements of a wearer's head. In certain embodiments, a motion sensoralso provides a means for determining motion of a wearer's head andprovides a new orientation and/or position of the head mounted displayto the computer subsystem in order to update augmentation graphicsdisplayed on a display screen of the head mounted display. In certainembodiments, a motion sensor is configured to provide a less than 2 cm(e.g., less than 1 cm) drift in the position of augmentation graphicsfrom their baseline position (i.e., as determined by the registration ofthe augmented reality navigation system to a patient's anatomy), asmeasured in an open loop. In certain embodiments, the motion sensorcoupled with a detector in an augmented reality navigation systemprovides optically calibrated (i.e., closed loop) sub-millimeter driftper hour. In certain embodiments, a motion sensor is used to record userinput used for making selections (e.g., a head nod or shake).

In certain embodiments, an augmented reality navigation system comprisesa microphone. The microphone may be disposed on the head mounteddisplay. The microphone may be used to receive surgeon commands (i.e.,verbal commands) for use in controlling the augmented reality navigationsystem. For example, verbal commands may serve as user input for makinga selection, modify or update settings or graphics of the displayscreen, or other similar tasks. For example, verbal commands may be usedto rearrange virtual display panels displayed on a display screen of ahead mounted display or to change the navigational information displayedon a virtual display. Similarly, graphics overlaid may be modified byverbal commands (e.g., to change brightness, color, or data source forthe overlaid graphics).

In some embodiments, in addition to or in place of verbal commands orcommands input by motion of a motion sensor, gestures made by a surgeonmay be used to signal selections or control an augmented realitynavigation system. For example, in certain embodiments, a surgeon canmake swipe, click, resize, or refresh type gestures (e.g., similar tothose used with smartphones or other touch control devices) to controlthe augmented reality navigation system. Such gestures may be made in amanner that is detectable by a detector of the augmented realitynavigation system. For example, gestures may be detected usingimage-recognition type procedures (e.g., when a detector is an opticalcamera). For example, a surgeon may wear or hold a fiducial detectableby the augmented reality navigation system when making a gesture suchthat the gesture is detected based on motion of the fiducial. In certainembodiments, alternatively or additionally, an auxiliary mechanicalinput (e.g., foot pedal) may be used as an input (e.g., selection)device.

The augmented reality navigation systems disclosed herein comprise acomputer subsystem. The computer subsystem, inter alia, processes (e.g.,renders) augmentation graphics for display on a display screen of anaugmented reality navigation system. The computer subsystem can beremote. For example, a computer subsystem can be connected to a headmounted display through a cable with a quick disconnect. Such a cablecan also provide power to components of a head mounted display (e.g., adisplay screen). In certain embodiments, a computer subsystem isdisposed partially or entirely on a head mounted display (e.g., thatoperates using battery power). In certain embodiments, a computersubsystem is configured to receive navigation input data that comprisesnavigational information from a robotic surgical system. For example, atrajectory stored in the robotic surgical system and/or coordinatesstored in the robotic surgical system (e.g., recorded using the roboticsurgical system). In certain embodiments, a computer subsystem cancontrol a robotic surgical system using output (such as trajectory dataoutput) from an augmented reality navigation system.

In certain embodiments, a computer subsystem is configured to assist inpre-operative and/or intra-operative planning (e.g., trajectoryplanning). In certain embodiments, the computer subsystem is configuredto perform trajectory planning using a pointer tool and/or trajectoryselection guidance augmentation graphic (e.g., crosshair) displayed on adisplay screen. In certain embodiments, a computer subsystem comprises amodel module that, inter alia, stores, accesses, and/or updates a modelof patient anatomy (e.g., derived from medical image data). In certainembodiments, a computer subsystem comprises a coordinate module that,inter alia, creates, stores, accesses, and/or updates a referencecoordinate system used for navigation. Such a reference coordinatesystem may be defined during a registration method (e.g., at the startof a surgical procedure). A coordinate module may be used to performreregistration that may happen continuously or periodically during asurgical procedure, wherein the registration may be performed using theaugmented reality navigation system.

In certain embodiments, an augmented reality navigation system operatescooperatively with an auxiliary navigation subsystem. For example, anauxiliary detector in a surgical environment may be used to provideadditional detection of real-world features in a surgical environment(in addition to those detected by a detector disposed on a head mounteddisplay). In this way, in certain embodiments, an augmented realitynavigation system is couple with an existing navigation system used inconjunction with a robot surgical system in order to navigate during asurgical procedure. For example, a registration performed by anaugmented reality navigation system can be compared to a registrationperformed by an auxiliary detector to determine accuracy of theregistration being used by the augmented reality navigation systemduring navigation of a surgical procedure. A computer subsystem canperform a reregistration to minimize error between a registrationaccording to an augmented reality navigation system and an auxiliarydetector. When an auxiliary detector is used, registration precision andtracking during a procedure may be improved, for example, due to eachdetector detecting real-world features that are otherwise obfuscatedfrom the other detector(s). In certain embodiments, an auxiliarydetector is a detector disposed on a head mounted display of a secondaugmented reality navigation system. An auxiliary detector, whenpresent, may be used to detect one or more fiducials disposed on a headmounted display (e.g., to assist in registration, jitter correctionand/or drift correction).

In certain embodiments, two or more augmented reality navigation systemsare used cooperatively and/or conjunctively (e.g., simultaneously). Forexample, two surgeons may each wear a head mounted display during asurgical procedure. In certain embodiments, navigational informationfrom one head mounted display (e.g., detector input data from a detectormounted to the head mounted display) may be provided to the other headmounted display, for example, in order to share a common registration orprovide a video input feed for display on a display screen of one headmounted display from a detector of another head mounted display. Incertain embodiments, a video input feed is provided to an externalmonitor (e.g., on a nearby wall or in a remote location, such as a classroom) for view by other persons (e.g., in the same surgicalenvironment), either in addition to or in alternative to being providedto a second augmented reality navigation system. In certain embodiments,fiducials disposed on each head mounted display assist in co-registeringtwo augmented reality navigation systems used in a single surgicalenvironment. Two or more augmented reality navigation systems may sharea common computer subsystem (e.g., each be connected by its own cable toa common computer subsystem).

Exemplary Augmented Reality Navigation Systems

FIG. 1 illustrates an augmented reality navigation system 100 (alsoreferred to as “HMD 100”) configured according to some embodiments ofthe present disclosure. Referring to FIG. 1, the HMD 100 includes asemitransparent display screen 110 connected to a display module thatprocesses and displays augmentation graphics (e.g., video and otherimages) on the display screen 110 (e.g., a LCD display, a reflectivescreen on which the display module projects images, etc.) for viewing bya user. The display module may be within a housing 118 of the HMD 100 ormay be contained within a communicatively connected computer subsystem.

In the illustrated embodiment, the HMD 100 is mounted to a headband 120and positioned so that the display screen 110 extends within theperipheral vision of the user. The housing 118 encloses electroniccomponents that display information on the display screen 110 and mayoperate in combination with a remote but communicatively connectedcomputer equipment and/or with computer equipment integrated within thehousing 118 to sense and interpret movement of the head, sense andinterpret gestures made by a user's hands or other objects, and/or senseand interpret voice commands by the user. The display screen 110 canprovide a monocular see-through display or a stereo set of see-throughdisplays so that the user can view information displayed on the displaywhile looking through the display to view other objects. The headband120 may include a headlamp, camera, or other apparatus that can be wornon a user's head.

The user is illustrated as wearing glasses 150 that includethrough-the-lens (TTL) loupes 152, protruding from lenses of the glasses150, that provide magnified viewing to the user. The display screen 110extends downward from the housing 118 and is positionable by the user tobe in the user's field-of-view or immediately adjacent to the TTL loupes152 within the user's peripheral vision. The display screen 110 can be asee-through display device allowing the user to see video superimposedon the environment seen through the display screen 110.

The TTL loupes 152 may not be included in the HMD 100 when the displayscreen 110 is configured to be in the direct line-of-sight of the user.Alternatively, the display screen 110 can be positioned adjacent to theTTL loupes 152 so that the user can make a minor upward shift in eyeline-of-sight from looking through the TTL loupes 152 to instead viewinformation displayed on the display screen 110. In some embodiments thedisplay screen 110 may be incorporated within one or both TTL loupes 152so that the user can look through the TTL loupes 152 to view graphicalimages super-imposed on objects within the FOV of the TTL loupe 152. TheHMD 100 may be configured to be attachable to any type of eyeglassframes, including prescription glasses, protective glasses, frameswithout lenses, transparent or protective shields, etc.

The display screen 110 can be moved by a user to a location providingconvenient visual reference through a two-arm friction joint linkage 112that provides telescopic and up-and-down adjustment of location of thehousing 118. A ball-and-socket joint 114 is connected between thelinkage 112 and the housing 118 to provide planar adjustment for thedisplay screen 110. A pivot joint 116 connected between theball-and-socket joint 114 and the housing 118 allows the user to pivotthe housing 118 and connected display screen 110. The display screen 110can thereby be flipped-up outside the user's peripheral vision when notbeing used.

The HMD 100 may include a motion sensor such as an inertial sensor orone or more other sensors, such as a gyroscope, accelerometer (e.g., amulti-axis accelerometer), and/or magnetometer that output a signalindicating a measurement of movement or static orientation of the user'shead while wearing the HMD 100. For example, the motion sensor mayoutput a head motion signal that indicates yaw (i.e., rotation of theuser's head left or right), pitch (i.e., rotation of the user's head upor down), and/or roll (i.e., side-to-side tilt of the user's head). Thesensors may be spaced apart on the headband 120 or enclosed within thehousing 118.

The HMD 100 may include an optical camera (e.g., acting as the detectoror in addition to other detector(s)) facing away from the user thatoutputs video and/or other images for processing and relay to other HMDs100 worn by other personnel assisting with the procedure, to otherdisplay devices, and/or to a video server for storage. For example, thecamera may be configured to be aligned with the user's line-of-sightwhen the user has adjusted the display screen 110 to be comfortablyviewed by the user. When more than one camera is connected to the HMD100, video streams from the cameras can be provided to an operationalfunction that estimates distance to an object viewed by the cameras. Theoperational function can include triangulation of distance to the objectbased on angular offset of the object viewed in the video streams and aknown distance between the cameras.

The camera (or another detector) may be connected to a gestureinterpretation module configured to sense gestures made by a user'shands or other objects, recognize a gesture as corresponding to one of aplurality of defined commands, and trigger operation of the command. TheHMD 100 may include a microphone connected to a voice interpretationmodule configured to recognize a received voice command as correspondingto one of a plurality of defined voice commands, and trigger operationof the command.

The headband 120 may have a plurality of attachment points whereinertial sensor(s), detector(s) (e.g., optical camera(s)), microphone,etc. can be releasably attached. Some of the attachment points may haverigid supporting structures between them to maintain a defined physicalalignment between the attached inertial sensors, detectors etc.

FIG. 2 illustrates a side view of another exemplary HMD 200 with adisplay screen 210 and electronic components 214 (shown without ahousing) which are configured according to some embodiments of thepresent disclosure. The display screen 210 extends downward from theelectronic components 214 to be in the user's line-of-sight orimmediately adjacent TTL loupes 152 within the user's peripheral vision.The electronic components 214 are connected to the headband 120 via apivot 212 allowing the electronic components 214 and connected displayscreen 210 to be flipped-down to a deployed position as shown in FIG. 2and flipped-up to a stored position when the user does not desire toview the display screen 210.

FIG. 3 illustrates another exemplary HMD 300 configured according tosome embodiments of the present disclosure. The HMD 300 includes adisplay screen illustrated behind a protective shield 310 that extendsdownward from a housing 318 enclosing electronic components. The displayscreen and/or the protective shield 310 may include a coating thatprovides variable contrast to enhance viewability of displayedinformation while subject to a range of ambient brightness. Theprotective shield 310 may provide a variable focal point (diopter). Theprotective shield 310 can be flipped from a stored up-position to aprotective down-position (as shown in FIG. 3) to cover an outsidesurface of the display screen that faces a patient and function toprotect the display screen from fluids and other materials occurringduring a procedure. The display screen can be moved by a user through atwo-arm friction-joint linkage 312 that provides telescopic andup-and-down adjustment of location of the housing 318 to enable a userto position the display screen at a location providing convenient visualreference. A ball-and-socket joint 316 is connected between the linkage312 and the housing 118 to provide planar adjustment for the displayscreen. The linkage 312 is connected to the headband 120 through a pivotjoint 314 to allow the user to flip the housing 318 and connecteddisplay screen up and down. The display screen can thereby be flipped-upoutside the user's line-of-sight or the user's peripheral vision whennot being used.

FIG. 4 illustrates another exemplary HMD 400 configured according tosome embodiments of the present disclosure. The HMD 400 includes adisplay screen 410 that extends downward from a housing 418 enclosingelectronic components. The display screen 410 and housing 418 areconnected to a ball-and-socket joint 416 which provides planaradjustment for the display screen 410. The ball-and-socket joint 416 isconnected to a pivot 414 that allows the housing 418 and connecteddisplay screen 410 to be pivoted up and down, so that the display screen410 can be flipped-up outside the user's line-of-sight or the user'speripheral vision when not being used. The pivot 414 is connected to asliding arm 412 that connects to the headband 120. The sliding arm 412provides telescoping adjustment to allow user placement of the displayscreen 410 a desired distance from an eye.

FIG. 5 illustrates a front view of the HMD 400 of FIG. 4 with thehousing 418 removed to expose printed circuit boards (PCBs) 450 whichoperationally connect electronic components mounted thereon (e.g., adisplay screen, a detector, and optionally a microphone and motionsensor). Some of the electronic components are used to display images onthe display screen 410, and may operate in combination with integratedor remote computer equipment to sense and interpret movement of thehead, sense and interpret gestures made by a user's hands, eyes, orother objects, and/or sense and interpret voice commands by the user. Insome embodiments, the PCBs 450 are tilted at a defined non-zero anglerelative to vertical to reduce the profile cross-section of the housing418. For example, the PCBs 450 can extend generally diagonally acrossthe housing 418.

FIG. 6 illustrates another HMD 500 having a single display screenconnectable to an eyeglass frame to provide monocular viewing by theuser. FIG. 7 illustrates another HMD 502 including a pair of displayscreens that are connectable to opposite sides of an eyeglass frame toprovide binocular viewing. Although the display screens in FIGS. 6 and 7are shown as being opaque, they may instead allow a user to see throughthe display screen while viewing information displayed thereon.

FIG. 8 shows certain exemplary functionality of certain augmentedreality navigation systems. An augmented reality navigation systemallows a surgeon or other user to see one or more virtual displays(several in the example shown in FIG. 8) of different medicalinformation without looking away from the surgical site and focusing faraway to view physical monitors that may be mounted across the surgicalenvironment or elsewhere adjacent to a patient. In some embodiments,three operational “modes” of the virtual displays are selectivelyactivated based upon pitch of the surgeon's head and the correspondingviewing line-of-sight of the user. The three operations may beseparately activated by increasing pitch angle of an HMD 750 throughthree corresponding ranges of viewing angles, such as low (directly atthe surgical space), medium, high (horizontal eye-level). In certainembodiments, the viewing angle of the surgeon can be determined from thehead motion signal output by a motion sensor of HMD 750.

A full-screen operational mode may be triggered when it is determined(e.g., by a motion sensor) that a surgeon is looking down at anoperation site, which may be determined by when the pitch is below afirst pitch threshold (e.g., about −45°). The first pitch threshold maybe defined and/or adjusted by the surgeon based on a voice command,entered through a physical user interface, etc. In the full-screenoperational mode, a defined one of the video streams (e.g., a primaryvideo stream received via an HDMI channel) is displayed, usingaugmentation graphics, full screen through a display screen 752 of theHMD 750. A surgeon's corresponding preference settings may be saved in aconfiguration file stored in the memory 630 of a computer subsystem withan identifier for the surgeon, so that the surgeon's preferred settingscan be automatically retrieved upon recognition of the surgeon (e.g.,via a login process through the computer equipment 620).

FIG. 9 is a block diagram of electronic components of an exemplarycomputer subsystem, in accordance with certain embodiments of augmentedreality navigation systems disclosed herein, that includes modules 600for processing data input from and output to a head mounted display,computer processing equipment 620, and a surgical video server 650(e.g., that provides input data streams). The video server 650 can beconnected via a data network 640 to a patient database 642, imagingequipment 644, and other electronic equipment 646, for example. The HMD600 may correspond to any of the HMDs of FIGS. 1-7, for example.Although the computer equipment 620 is illustrated as being separatefrom the HMD 600, some or all of the operations disclosed herein asbeing performed by the computer equipment 620 may additionally oralternatively be performed by one or more processors residing within theHMD 600. Similarly, some of the operations disclosed herein as beingperformed by the HMD 600 may additionally or alternatively be performedby one or more processors residing within the computer equipment 620.

The video server 650 can receive, store, and route information, videostreams between a patient database 642, imaging equipment 644, and otherelectronic equipment 646 and the HMD 600. As used herein, a video streamcan include any type of information that can be provided to a displaydevice for display, including without limitation a still image (e.g.,digital photo), a sequence of still images, and video having framesprovided at a defined frame rate. A video stream may comprisenavigational information. Imaging equipment 644 may include endoscopecameras, magnetic resonance imaging equipment, computed tomographyscanning equipment, three-dimensional ultrasound equipment, endoscopicequipment, and/or computer modeling equipment which can generatemultidimensional (e.g., 3D) model based on combining images from imagingequipment. A patient database 642 can retrievably store informationrelating to a patient's medical history, and may store patient imagesfrom earlier procedures conducted via the imaging equipment 644. Otherequipment 646 may provide information relating to real-time monitoringof a patient, including, for example, hemodynamic, respiratory, andelectrophysiological signals.

Computer equipment 620 operationally interfaces HMD 600 to the videoserver 650. Computer equipment 620 includes a video capture card 622that can simultaneously receive a plurality (N) of video streams andinformation (e.g., textual descriptions, audio signals, etc.) from videoserver 650 and/or directly from imaging equipment 644, patient database642, and/or the other equipment 646. Computer equipment 620 maycommunicate with video server 650, HMD 600, and other equipment of thesystem via a wireless and/or wired network interface 628 using anyappropriate communication medium, including but not limited to awireless air interface (e.g., 3GPP Long Term Evolution (LTE), WLAN (IEEE802.11), WiMax, etc.), wireline, optical fiber cable, or any combinationthereof. In the example embodiment of FIG. 9, the video capture card 622simultaneously receives up to 4 video streams via 4 HDMI interfaces. Insome embodiments, HMD 600 is communicatively connected to computerequipment 620 via an HDMI cable, a USB or RS 422 cable connected to themotion sensor 604 and/or gesture sensor 602, and a USB 3.0 or firewirecable connected to the camera 610. Gesture sensor 602 may be motionsensor 604, detector 610, or a distinct sensor for detecting andprocessing gestures. A microphone 612 can be connected to the computerequipment 620. The video and/or sensor signaling may alternatively becommunicated between HMD 600 and computer equipment 620 through awireless air interface, such as network interface 628.

HMD 600 includes a display module 606 that processes and displays videoand other images on the display screen 608 for viewing by a user. Thevideo streams received by the video capture card 622 are processed by agraphics processing unit (GPU) 638, conditioned by a display driver 614,and provided to the display module 606 for display on the display screen608. A symbol generator 624 may add graphical indicia and/or textualinformation to the video stream(s) provided to the HMD 600 based oninformation received from the video server 650 (e.g., via the patientdatabase 642).

Display driver 614 may reside in the computer equipment 620 or the HMD600. In some embodiments, display driver 614 receives video via a HDMIinterface from GPU 638, and converts a digital video signal to an analogvideo signal which is output as low-voltage differential signaling(LVDS) to display module 606. Display driver 614 may also provide powerand/or other signaling to display module 606 via an LED drive signal.

HMD 600 can include a detector (e.g., optical camera) 610, or aplurality of detectors 610, facing away from a wearer that outputs videoand/or other images via a wireless and/or wired network interface 628,illustrated as a HDMI cable in FIG. 9, to GPU 638 for processing andrelay to video server 650 for storage and possible further relay toother HMDs 600 worn by other personnel assisting with a procedure. Forexample, detector 610 may be configured to be aligned with a wearer'sline-of-sight when the wearer has adjusted the display screen 608 to becomfortably viewed by the wearer. In certain embodiments, a video signalfrom detector 610 can be processed through computer equipment 620 andprovided to video server 650 for recording what the wearer is viewingduring a procedure and/or can be provided as a real-time video stream toother HMDs 600 worn by personnel assisting with the procedure so thatthe personnel can observe what the user is seeing. A video signal fromdetector 610 may be augmented by symbol generator 624 with one or moredesignation symbols such that augmentation graphics displayed on adisplay screen (e.g., of an HMD worn by another wearer) comprise boththe video signal and one or more symbols. Augmentation graphicscomprising one or more symbols may, for example, identify the firstwearer as the source of the video stream and/or be added to a videostream by a wearer to identify observed features, such as a patient'sanatomy.

The HMD 600 may include a motion sensor 604 and/or a gesture sensor 602.Motion sensor 604 may be an inertial motion unit (IMU), gyroscope,accelerometer (e.g., a multi-axis accelerometer), and/or tilt sensorthat outputs a head motion signal indicating a measurement of movementof the user's head while wearing the HMD 600. Motion sensor 604 may bepowered by computer equipment 620 and may output a head motion signalvia a communication interface, such as a RS-422 serial digitalinterface. For example, motion sensor 604 may output a head motionsignal that indicates yaw movement (i.e., rotation of the user's headleft or right) and/or indicates pitch movement (i.e., rotation of theuser's head up or down).

Motion sensor 604 may be a sourceless orientation sensor. A head motionsignal output by a motion sensor may be processed by HMD 600 and/or bycomputer equipment 620 to compensate for drift error introduced by themotion sensor 604. In some embodiments, one directional reference (e.g.,yaw) component of a head motion signal is corrected toward zeroresponsive to another reference component (e.g., pitch) of the headmotion signal being within a threshold offset of a defined value. Forexample, yaw drift error in the head motion signal can be determinedbased on monitoring yaw values of the motion signal while the user islooking down at a defined pitch (e.g., pitch being within a thresholdrange of a defined value) to align the user's eyes with an object (e.g.,when a surgeon repetitively looks down to view a surgical site of apatient). In some embodiments, responsive to the pitch component of thehead motion signal indicating that a surgeon is looking down for atleast a threshold time that is indicative of the surgeon visuallyconcentrating on a surgical site, computer equipment 620 assumes thatHMD 600 is stabilized along the yaw axis and computes yaw drift errorbased on measured change in the yaw component over a defined timeinterval. The head motion signal is then compensated to remove thedetermined yaw drift error. In some embodiments, computer equipment 620measures drift in the yaw component of a head motion signal while astatic image is displayed on the display screen, assuming that thesurgeon's head is stabilized along the yaw axis, and then compensatesthe head motion signal to remove the measured drift in the yawcomponent.

A head motion signal may be processed by HMD 600 and/or by computerequipment 620 to identify an origin for one or more directionalreference components from which movement is referenced. For example, anorigin location from which yaw is measured may be identified based on anaverage (e.g., median or mode) of a yaw component of the head motionsignal during times when the user is looking down at a defined pitch toalign the user's eyes with an object (e.g., surgeon looking down to viewa surgical site).

The directional reference (e.g., pitch or yaw) of a head motion signal,which is defined to trigger compensation for drift error and/or which isdefined as a reference origin for movement measurement, may beidentified based on the user maintaining a substantially constantorientation of HMD 600 for a threshold time (e.g., dwell time). Forexample, when a surgeon has maintained a relatively constant headposition while viewing a surgical site of a patient for a thresholdtime, the directional reference (e.g., pitch or yaw) of the head motionsignal during that dwell time can be used as a basis for compensatingfor drift error and/or setting as a reference origin for display ofvirtual display panels (e.g., as illustrated in FIG. 8) and/or otheraugmentation graphics (e.g., that appear overlaid over a physicalobject, such as patient anatomy). In some embodiments, a head motionsignal may be processed by the HMD 600 and/or by the computer equipment620 to estimate gyroscope bias(es) giving rise to yaw drift and/or pitchdrift accumulating over time based on pseudo-measurements of the yawand/or the pitch provided by the head motion signal which is expected tobe nearly zero each time the surgeon looks down at the same surgicalsite and steadies the head to center the line-of-sight at a samelocation on the patient.

Gesture sensor 602 may include any type of sensor that can sense agesture made by a user. In a surgical environment, use of a gesturesensor 602 to receive a gesture-based command from a surgeon or other ORpersonnel can be advantageous because it avoids a need for the user totouch a non-sterile surface of the HMD 600 or other device. The gesturesensor 602 may be or include detector 610 which outputs signal (e.g.,RGB-D video) displaying movement of a user's hand, fingers, arms orother objects moved by the user along a pathway that the user knows willdefine a command identifiable by an operational surgical program (OSP)632 and/or another component of the system. Detector 610 or anothercamera may be directed toward one of the user's eyes to identify a dwelltime of the eye, blink timing, and/or movement of the eye to generate acommand from the user to control what is displayed on the display screen608.

Gesture sensor 602 may alternatively or additionally include one or morephotoelectric motion and/or proximity sensors. In some embodiments,gesture sensor 602 has one or more infrared emitters and one or more ofphotodiodes (e.g., at least a portion of which may additionally functionas a detector for navigation). For example, adjacent pairs of aninfrared emitter and a photodiode can be spaced apart and arranged toform a directional array facing outward from a housing of HMD 600 tosense presence of a user's hand adjacent to the array and/or to sense adirection of movement as the user's hand is moved across the array. Auser may, for example, swipe a hand in a first direction across thearray (without touching the housing) to input a first type of gesturerecognized by OSP 632 processed by processor 626 which triggers a firsttype of operation by OSP 632, swipe the hand in a second direction aboutopposite to the first direction across the array to input a second typeof gesture recognized by OSP 632 which triggers a second type ofoperation by OSP 632, swipe the hand in a third direction aboutperpendicular to the first direction across the array to input a thirdtype of gesture recognized by OSP 632 which triggers a third type ofoperation by OSP 632, and so on with other directions of movement beingidentifiable as other types of gestures provided by the user to triggerother types of operations by OSP 632.

In some embodiments, gesture sensor 602 includes an ultrasonic echoranging transducer that senses signal echo reflections from a user'shand and outputs a signal to the processor 626 which identifies gesturesformed by movement of the hand. In some embodiments, gesture sensor 602includes a capacitive sensor that senses presence of a user's handthrough capacitive coupling between a charge plate and the user's hand.A plurality of capacitive sensors may be spaced apart to form gesturesensor 602 and configured to sense a direction of movement of the user'shand relative to the array of charge plates (e.g., sense an order withwhich plates experienced increased coupling to the user's hand).Different sensed directions of movement can be interpreted by OSP 632and/or another component of the system as representing differentcommands selected by the user for operation.

HMD 600 can include a microphone 612 configured to receive voicecommands from a user. The processor 626 executing OSP 632 and/or anothercomponent of the system can be configured to recognize a received voicecommand as corresponding to one of a plurality of defined voicecommands, and trigger operation of a command corresponding to therecognized voice command to control information (e.g., navigationalinformation) displayed on a display screen (i.e., via augmentationgraphics).

In some embodiments, video signal from detector 610 is displayed ondisplay device 608 of an HMD 600, and symbol generator 624 incombination with OSP 632 processed by the processor 626 may operate todisplay trajectory selection guidance augmentation graphics (e.g.,reticle such as crosshairs) that can be positioned within a plane of thevideo stream by a surgeon (e.g., responsive to recognition of voicecommands via a microphone 612 or to gestures via gesture sensor 602). Inthis manner, a surgeon may orient his or her natural field of view to asurgical site, steer a trajectory selection guidance augmentationgraphic to be aligned with a point-of-interest (e.g., in a patient'sanatomy), and trigger capture of data that represents a position and/ororientation (e.g., of a trajectory) of interest that is indicated by thetrajectory selection guidance augmentation graphic. For example, usingan augmented reality navigation system registered to a patient anatomy,a trajectory can be planned by orienting and/or positioning a trajectoryselection guidance augmentation graphic (e.g., through movement of asurgeon's head or movement of the augmentation graphic) and providing auser input that captures a position and/or orientation indicated by thetrajectory selection guidance augmentation graphic. A trajectoryselection guidance augmentation graphic may be fixed in position on adisplay screen or may be moveable. In some embodiments, a pointer toolis used to determine such a point of interest.

FIG. 10 is a block diagram of components of an augmented realitysurgical system that include a position tracking system 810 (e.g., oneor more detectors mounted on a head mounted display and, optionally,cameras spaced apart in the operating room) that track the location of apatient's anatomy, surgical tool 800 and/or surgical apparatus (e.g., animplant or other object used as part of a surgical procedure).Generally, any object comprising a real-world feature capable of beingidentified by a detector (e.g., mounted on a head mounted display) canbe tracked and used in navigation with an augmented reality navigationsystem in accordance with embodiments disclosed herein. In certainembodiments, computer subsystem 820 uses patient data from imagingequipment 830 to generate a two dimensional (2D) or three dimensional(3D) model. Imaging equipment 830 may include, without limitation, x-rayequipment, endoscope cameras, magnetic resonance imaging equipment,computed tomography scanning equipment, three-dimensional ultrasoundequipment, endoscopic equipment, and/or computer modeling equipmentwhich can generate a multidimensional (e.g., 2D or 3D) model of atargeted site of a patient. The patient data can include real-time feedsand/or earlier stored data from imaging equipment 830, and may includean anatomical database specific for the particular patient or moregenerally for humans.

The model can include representations of real-world features that assistwith performing correlations (e.g., registrations) between virtuallocations of the representations of real-world features in the patientmodel and physical locations of the real-world features (e.g., on thepatient's body). Computer subsystem 820 can use (i) present locations ofHMD 100, surgical site 804, and surgical tool 800 and/or surgicalapparatus 802 obtained by position tracking system 810 by detecting oneor more real-world features and (ii) the real-world featurerepresentations contained in a patient model to transform the patientmodel to a present perspective view of a wearer of HMD 100. Some or allof the transformed patient model can then be displayed on a displayscreen of HMD 100 using augmentation graphics, for example, to providethe surgeon with an augmentation graphical overlay that is preciselyoriented and scaled on the surgical site 804 or other target location ona patient.

Computer subsystem 820 may augmentation graphics representing a patientmodel, a surgical tool and/or a surgical apparatus on a display screen110 of an HMD 100. Augmentation graphics may be portions of a surgicaltool and/or a surgical apparatus that are otherwise not viewable by asurgeon during at least a portion of a surgical procedure (e.g., arecovered by patient anatomy such as a patient's skin). Computer subsystem820 may additionally be configured to animate movement of a displayedpatient mode, tool and/or surgical apparatus (e.g., to illustrate aplanned procedure relative to a defined location of the surgical site804 or other target location on the patient's body). The HMD 100 may becommunicatively connected to the computer subsystem 820 through awireless transceiver and/or wired network interface.

Computer subsystem 820 may compare patterns of objects in a detectorinput signal (e.g., video stream) from a detector (e.g., camera) on theHMD 100 to patterns of real-world features in the patient model toidentify levels of correspondence, and may control transformation of thepatient model responsive to identifying a threshold level ofcorrespondence between the compared objects. For example, real-timevideo captured by an HMD-mounted camera during surgery of a patient maybe processed by computer subsystem 820 and compared to video captured byone or more other sources, e.g., an auxiliary detector of imagingequipment 830. The pattern matching may be constrained tocharacteristics of an object or a set of objects defined by a surgeon asbeing relevant to a present procedure.

The computer subsystem 820 can control transformation of a patient modelfor display on the display screen 110 using augmentation graphics basedon the pattern matching (e.g., with or without comparing to a detectorinput signal from auxiliary imaging equipment). The computer subsystem820 may display on the display screen 110 an indicia (e.g., a crosshairor color marker) aligned with an identified object within the video fromthe HMD camera to assist the surgeon with identifying the correspondinglocation on the patient. In one embodiment, the computer subsystem 820displays a graphical indicia on the display screen 110 aligned with oneof the anatomical objects displayed on the display screen 110 from therotated and scaled three dimensional anatomical model responsive toidentifying a threshold level of correspondence between a pattern of theone of the anatomical objects and a pattern of one of the anatomicalobjects in the video stream from the video camera.

Computer subsystem 820 may similarly receive other data and videostreams from a patient database and other electronic equipment, whichcan be selectively displayed on one or more display screens of an HMD100 using augmentation graphics. As used herein, a video stream caninclude any type of information that can be provided to a display devicefor display, including without limitation a still image (e.g., digitalphoto), a sequence of still images, and video having frames provided ata defined frame rate. Computer subsystem 820 can retrieve patient healthinformation relating to a patient's medical history and data obtained byreal-time monitoring of a patient, including, for example, hemodynamic,respiratory, and electrophysiological signals. Such information can bedisplayed on a display screen using augmentation graphics and, moreover,can appear to be displayed on a virtual display screen and/or overlaidover an object in a surgical environment (e.g., patient anatomy orsurgical equipment).

In certain embodiments, computer subsystem 820 is physically separate(e.g., remote) from a connected HMD 100. In some embodiments, some orall of the operations disclosed herein as being performed by a computersubsystem 820 are additionally or alternatively performed by one or moreprocessors residing within an HMD 100. Likewise, in some embodiments,some or all of the operations disclosed herein as being performed by anHMD 100 are additionally or alternatively performed by one or moreprocessors residing within a computer subsystem 820.

FIG. 11 is a block diagram of electronic components in an augmentedreality surgical system according to some embodiments of the presentdisclosure. Referring to FIG. 11, the navigation system 810 can includeat least one detector 902 that tracks at least one of, for example,real-world features identifying and/or attached to a surgical table 904,real-world features identifying and/or attached to patient 906 (e.g.,adjacent to a surgical or other target site), real-world featuresidentifying and/or attached to a surgery tool and/or surgical apparatus(e.g., implant) and/or robotic surgical system 908, and, optionallyreal-world features attached to an HMD 910. In some embodiments, thedetectors used for navigation (e.g., tracking) are exclusively attachedto a HMD (i.e., no auxiliary detector(s) are used). In theseembodiments, the volume over which a robotic surgical system and apatient and, optionally, a patient model, are registered is defined by avolume at least temporarily within the field of view of the detectorsattached to the HMD. In some embodiments, auxiliary detector(s) notattached to a HMD are used for navigation. For example, in someembodiments, the at least one detector 902 includes a plurality ofcameras that are spaced apart at defined locations within an operatingroom and each having a field of view that can observe objects to betracked. In the illustrated example, the camera system 902 includes twosets of cameras spaced apart by a known distance and relativeorientation. The navigation system 810 may use, for example and withoutlimitation, active optical real-world features (e.g., light emittingsources) or passive optical real-world features (e.g., lightreflectors). Navigation system 810 may additionally or alternatively useelectromagnetic field or radiation based navigational tracking,ultrasonic based navigational tracking or the like (e.g., depending on atype of detector mounted to a head mounted display and/or a type ofauxiliary detector).

In some embodiments, positioning data of an HMD 100 can includenavigation coordinate system data determined from a location ofreal-world features attached to an HMD 910 and/or inertial coordinatesystem data from a motion sensor attached to the HMD. In someembodiment, the navigation coordinate system data and the inertialcoordinate system data can be compensated for initial calibration anddrift correction over time by a calibration module 912 and combined by afusion module 914 to output combined HMD position data. The calibrationcomponent and fusion component may be modules of a computer subsystem.

A relative positioning module 916 identifies the relative position andangular orientation of each of the tracked real-world features 904-910and the combined HMD position data. A relative positioning component maybe a component of a computer subsystem. The module 916 may performcoordinate transformations of relative coordinate systems of, forexample, a surgical table, a patient, a surgical tool (and/or a pointertool), and an HMD 100 to a unified (common) coordinate system. In someembodiments, a relative positioning module 916 outputs sight coordinatesdata, patient model coordinates data, and tool coordinates data to anaugmentation graphics generator module 918 and/or a robotic surgicalsystem. An augmentation graphics generator module 918 may be part of acomputer subsystem of an augmented reality navigation system. Sightcoordinates data can be generated based on the combined HMD positiondata transformed to the unified coordinate system. In some embodiments,a spatial position of an HMD (e.g., a position on the HMD, such as aposition of a detector mounted to the HMD) is taken to be an origin of aunified coordinate system and additional spatial coordinates determinedfrom various real-world features are registered to the HMD using theunified coordinate system with that position as the origin. Accordingly,in some embodiments, a registration is updated [e.g., continuously(e.g., at a certain refresh rate)] to account for movements of an HMDthroughout a surgical procedure that cause the position of the origin ofthe unified coordinate system in the physical space of a surgicalenvironment to be changed. Movement of an HMD may be determined by amotion sensor or a change in a fixed position real-world feature (e.g.,a real-world feature identified by or attached to a surgical table).Patient model coordinates data can be generated based on the positionand/or orientation of real-world features identified from or attached toa patient 906 transformed to the unified coordinate system. Toolcoordinates data can be generated based on a position and/or orientationof real-world features identified from or attached to a surgical tool908 transformed to the unified coordinate system. Tool coordinates datacan be generated based on a position and/or orientation of real-worldfeatures identified from or attached to a surgical tool 908 transformedto the unified coordinate system. In some embodiments, robotic surgicalsystem coordinates data and tool coordinates data are equivalent (i.e.,one coordinate system is used for a surgical tool and a robotic surgicalsystem simultaneously, for example, when the surgical tool is attachedto the robotic surgical system).

In certain embodiments, augmentation graphics generator module 918transforms (e.g., scales, rotates, translates) a patient model (e.g.,derived from medical image data) to a present perspective view of awearer of an HMD 100 (e.g., mapped to a corresponding object within theFOV of a display screen 110). Likewise, a model of, for example, arobotic surgical system, a surgical tool, a surgical apparatus, or atrajectory of the same, may be transformed by an augmentation graphicsgenerator module, in some embodiments. In some embodiments, a graphicsaugmentation generator module 918 may provide video generated based onthe transformed patient model to a display screen of an HMD 100 fordisplay as a visual model that is dynamically oriented and scaled as agraphical overlay on a surgical site 804 or elsewhere to a correspondinglocation on the patient where the wearer of the HMD 100 is presentlylooking and which contains a corresponding object which is modeled bythe patient model.

For example, in some embodiments, an augmentation graphics generatormodule 918 determines whether any portion of a patient's body ispresently within the field of view of what the surgeon sees through thedisplay screen 110 (e.g., using a detector mounted to the same HMD asthe display screen) that corresponds to any portion of the transformedpatient model. In some embodiments, when a portion of a transformed(e.g., scaled, translated, rotated) patient model corresponds to aportion of the patient's body within the surgeon's field of view throughthe display screen 110, the image generator 918 generates augmentationgraphics for display on the display screen 110 based on thecorresponding portion of the transformed patient model, whiletranslating and/or rotating the portion of the transformed patient modeland scaling size of the portion of the transformed patient model toprovide an accurately scaled graphical representation of the object thatwas, for example, imaged from the patient or modeled from another sourcesuch as an anatomical database.

Thus, for example, in some embodiments, when a surgeon's head is rotatedso that a portion of a patient's body having a bone that is modeledthrough CT imagery data becomes within the field of view of the displayscreen 110 (e.g., within a field of view of a detector mounted to thesame HMD as the display screen 110)), an augmentation graphics generatormodule 918 transforms a patient model of the bone to generate anaugmentation graphical representation of the bone that is displayed inthe display screen 110 as a graphical overlay that matches theorientation and size of the bone from the perspective of the surgeonas-if the surgeon could view the bone through intervening layers oftissue and/or organs. Likewise, in some embodiments, at least a portionof a surgical tool, at least a portion of a surgical apparatus, and/orat least a portion of a robotic surgical system (e.g., that is coveredby a patient's anatomy) can appear as a graphical overlay matching theorientation and size of the physical object to the surgeon usingaugmentation graphics, in some embodiments.

In the example illustration of block 920, a leg bone model that has beengenerated, e.g., based on a CT scan of the leg, is transformed anddisplayed on a display screen 110 to have an accurate orientation andsize (e.g., six degree of freedom positioning) relative to the leg bonewhen viewed augmentation graphics of the leg bone 922 superimposed on askin surface of the leg. In this example, the surgeon therefore sees theskin surface of the leg through the semitransparent display screen 110of the HMD 100 with an graphically illustrated representation of the legbone model overlaid thereon.

Although the augmentation graphics of the leg bone 922 are illustratedin FIG. 11 as being displayed in a superimposed position on a skinsurface of the leg, the augmentation graphics 922 can be displayed atother locations which may be controllable by a surgeon. The surgeon may,for example, select to have the graphical representation 922 displayedwith a defined offset distance above or below the leg. Moreover, thesurgeon may control the size of the displayed augmentation graphics 922relative to the leg. The surgeon may, for example, temporarily magnifyaugmentation graphics 922 to view certain details and then returnaugmentation graphics 922 to be scaled and aligned with the leg.Additionally, augmentation graphics 922 may be modified to appear to bedisplayed on a virtual display screen hovering near the surgical site.Augmentation graphics 922 may be modified to also display a portion of asurgical tool, at least a portion of a surgical apparatus, at least aportion of a robotic surgical system, and/or a trajectory of any of thesame, either overlaid over the patient's leg or in a virtual displayscreen.

Use of Augmented Reality Navigation Systems in Surgical Procedures

Augmented reality navigation systems in accordance with embodimentsdisclosed herein may be used in methods of performing surgicalprocedures in order to provide and/or assist in navigation during theprocedures. As such, navigation input data can be provided to a roboticsurgical system, obtained from a robotic surgical system, or both incertain embodiments. Generally, in methods disclosed herein, at leastone detector mounted on (e.g., attached to) a head mounted display isused in order to determine coordinate systems used for navigation basedon real-world features detected by the at least one detector. In someembodiments, an augmented reality navigation system displaysaugmentation graphics that show portions of a surgical tool, surgicalapparatus, and/or robotic surgical system otherwise hidden from thenatural field of view of a surgeon. The surgical tool may be attached toor inserted into a robotic surgical system (e.g., through a tool guideattached thereto). In some embodiments, an augmented reality navigationsystem receives navigation input data from a robotic surgical system(e.g., comprising a position and/or orientation of the robotic surgicalsystem, a model stored thereon or created therewith, and/or one or moretrajectories that could be followed by the robotic surgical system). Insome embodiments, an augmented reality navigation system is used inconjunction with a pointer tool in order to define a trajectory that arobotic surgical system can follow. The pointer tool may be attached toor inserted in the robotic surgical system. In some embodiments, anaugmented reality navigation system is configured to display atrajectory selection guidance augmentation graphic in order to use thegraphic to determine a trajectory that a robotic surgical system canfollow.

The methods and systems disclosed herein can be used with roboticsurgical systems to perform surgical procedures using a particularrobotic surgical system. In certain embodiments, portions or all of aparticular surgical procedure are performed without the use of a roboticsurgical system. For example, one or more surgical tools may bemanipulated in a “free-hand” manor by a surgeon, wherein the surgeonholds the tool(s) or the tool(s) are held by an apparatus not connectedto a robotic surgical system. Such free-hand manipulation may occursimultaneously with, before, after, or in place of any action performedby a robotic surgical system (e.g., automatically) or with theassistance of a robotic surgical system. In any such case, an augmentedreality navigation system in accordance with one or more embodiments ofthe present disclosure can be used to display navigational informationrelated to the free-hand manipulation of any surgical tool or implant,such as tracking and trajectory planning of such an implant or surgicaltool (e.g., with or without use of any additional and/or auxiliarynavigation subsystem). It will be appreciated by one of ordinary skillin the art that, in the following described exemplary methods, wherereference is made to a surgical tool attached to or inserted into arobotic surgical system, it may be possible for the augmented realitynavigation system being used in the particular exemplary method beingdescribe to likewise perform the same function for a tool being held bya surgeon.

Referring now to the block flow diagram shown in FIG. 12, exemplarymethod 1200 is a method of using an augmented reality navigation systemin accordance with some embodiments of the present disclosure. Exemplarymethod 1200 is a method whereby an augmented reality navigation systemis used to display augmentation graphics representing a surgical tool(e.g., attached to or inserted into a robotic surgical system) and/orits trajectory. It is understood that such an exemplary method can alsobe adapted to display augmentation graphics representing at least aportion of a surgical apparatus (e.g., implant) that is, for example,attached directly or indirectly to a robotic surgical system. Forexample, an implant such as a screw may be attached to drill bit that isattached to a robotic surgical system. The apparatus may comprise one ormore real-world features (e.g., a fiducial attached thereto). Thecomputational steps of exemplary method 1200 may be performed by acomputer subsystem of an augmented reality navigation system.

In step 1202, one or more detectors mounted on a head mounted display ofan augmented reality navigation system generate a detector input signalthat is received by a computer subsystem of the augmented realitynavigation system. The detector input signal represents a field of viewof the one or more detectors that comprises at least a portion of apatient anatomy (e.g., relevant to a surgical procedure to beperformed). In step 1204, a relative position and/or orientation for oneor more real-world features in the detector input signal (i.e., in thefield of view represented by the detector input signal) are determined.The relative position and/or orientation may be determined in real timeor may be taken as an absolute position and/or orientation representedin a unified coordinate system that resulted from a registration of anaugmented reality navigation system and a surgical environment (e.g., aregistration between the augmented reality navigation system and objectsin the surgical environment such as a patient, a surgical tool, and arobotic surgical system, for example). The detector input signal may bereceived by a computer subsystem from one or more detectors mounted to ahead mounted display of the augmented reality navigation system, one ormore detectors mounted to a head mounted display of a second augmentedreality navigation system, and/or one or more auxiliary detectorspositioned throughout a surgical environment. In step 1206, arepresentation of at least a portion of a surgical tool connected toand/or inserted into a robotic surgical system and/or a trajectory(e.g., actual or planned trajectory) of the surgical tool is generatedand/or accessed. The at least a portion of the surgical tool may behidden from the natural field of view of a surgeon using the augmentedreality navigation system. In step 1208, the representation is modified(e.g., scaled, translated, and/or rotated) by the relative positionand/or orientation of the one or more real-world features determined instep 1204. In step 1210, augmentation graphics corresponding to therepresentation are rendered and displayed on a display screen of thehead mounted display of the augmented reality navigation system.Optionally, at least a portion of the surgical tool not hidden from anatural field of view of a surgeon may also be displayed, such that theaugmentation graphics show a representation of a larger portion of thesurgical tool (e.g., the entire surgical tool) wherein some of the toolis hidden and some of the tool is not hidden. Such a method may also beused for surgical apparatus (e.g., implants). Likewise, the surgicaltool augmentation graphics can also be used to show a surgical tool (orportion thereof) that is not physically present (e.g., has not yet beeninserted into a robotic surgical system). Additionally, in someembodiments, a position of a surgical tool and/or its trajectory can bevisualized over a period of time using augmentation graphics such that asurgeon can watch how a tool will move along a trajectory during atleast a portion of a procedure.

The generating and/or accessing a representation step may comprisedetermining a relative position and/or orientation of one or morereal-world features corresponding to the surgical tool of interest. Forexample, the surgical tool may have a fiducial affixed thereto. Therepresentation may be a stored model (e.g., a CAD model or illustration)of the surgical tool imported into or accessed by the computer subsystem(e.g., provided to the computer subsystem by the robotic surgical systemas navigational information). Similarly, in some embodiments, atrajectory of a surgical tool that is otherwise hard to visualize (dueto interference from a patient's anatomy) may be displayed on a displayscreen of a head mounted display as surgical tool augmentation graphicssuch that the surgeon can view one or more planned trajectoriesintraoperatively to better visualize the one or more trajectories andtherefore better navigate a robotic surgical system. The trajectoriesfor the surgical tool may be provided to the computer subsystem by arobotic surgical system. For example, a surgeon may move a roboticsurgical system to a desired position and orientation and save acorresponding trajectory representation that is then later provided to aaugmented reality navigation system in generating and/or accessing step1204. The representation of at least a portion of a surgical tool and/ortrajectory that is displayed on a display screen in exemplary method1200 may be selected using input from a surgeon, for example, using agesture, motion, or signal input as described herein above.

Exemplary method 1200 includes optional steps 1212 and 1214. Either orboth of the optional steps may be a part of a method. In optional step1212, a haptic object comprising the trajectory represented by thesurgical tool augmentation graphics displayed in step 1210 is defined.The haptic object may be determined by the augmented reality navigationsystem (e.g., using relative position(s) and/or orientation(s) ofreal-world features detected by a detector of the system) and providedto a robotic surgical system by the computer subsystem of the augmentedreality navigation system. The robotic surgical system may then beconfined in its motion such that at least a portion of a surgical toolconnected to or inserted into the robotic surgical system (e.g., thesurgical tool corresponding to the representation generated and/oraccessed in step 1206) cannot move outside of the haptic object (e.g.,due to haptic feedback provided to a surgeon operating the roboticsurgical system). In optional step 1214, output from the augmentedreality navigation system causes the robotic surgical system toautomatically align with the represented trajectory and/or automaticallymove along the trajectory. In some embodiments, representations ofmultiple trajectories may be displayed simultaneously and a surgeon mayselect one of the represented trajectories to have the robotic surgicalsystem align to (e.g., using a pointer tool). In some embodiments,whether one or multiple representations of trajectories are displayed,the augmentation graphics used may appear overlaid over a patientanatomy to give a surgeon an accurate view of how the trajectoryintersects a patient's anatomy. In some embodiments, portions of asurgical tool and/or its trajectories appear on a virtual display screenviewed through a display screen of a head mounted display (e.g., inspatial correspondence with a model of patient anatomy such that thetrajectory representation, portion of a surgical tool, and model appearto have the same relative position and orientation as physical reality).For example, a virtual display screen may appear as panel A or panel Bof FIG. 15. Overlaid augmentation graphics may appear as the sketch inFIG. 16 illustrates.

Referring now to the block flow diagram shown in FIG. 13, exemplarymethod 1300 is a method of using an augmented reality navigation systemin accordance with some embodiments of the present disclosure. Exemplarymethod 1300 is a method whereby an augmentation reality navigationsystem uses a trajectory selection guidance augmentation graphic (e.g.,a crosshair) to assist in planning and/or updating one or moretrajectories corresponding to a surgical procedure. The computationalsteps of exemplary method 1300 may be performed by a computer subsystemof an augmented reality navigation system.

In step 1302, a trajectory selection augmentation graphic (e.g.,crosshair) is displayed on a display screen of an augmented realitynavigation system. The trajectory selection guidance augmentationgraphic may appear fixed in position on the display screen (e.g., suchthat motion of the associated head mounted display does not result inthe augmentation graphic moving position in a surgeon's field of view).In some embodiments, a trajectory selection guidance augmentationgraphic is positionable (e.g., pre- and/or intra-operatively) by asurgeon. For example, a surgeon may prefer the augmentation graphic toappear in a certain location (e.g., center or corner of a displayscreen) and may position it accordingly using a selection input. In step1304, a detector input signal is received where the detector inputsignal corresponds to a field of view of one or more detectors connectedto a head mounted display of an augmented reality navigation system,wherein the field of view comprises at least a portion of a patient'sanatomy.

In step 1306, a relative position and/or orientation of one or morereal-world features (i.e., in the field of view represented by thedetector input signal) are determined based on a detector input signalfrom one or more detectors connected to a head mounted display. Therelative position and/or orientation may be determined in real time ormay be taken as an absolute position and/or orientation represented in aunified coordinate system that resulted from a registration of anaugmented reality navigation system and a surgical environment (e.g., aregistration between the augmented reality navigation system and objectsin the surgical environment such as a patient, a surgical tool, and arobotic surgical system, for example). The detector input signal may bereceived by a computer subsystem from one or more detectors mounted to ahead mounted display of the augmented reality navigation system, one ormore detectors mounted to a head mounted display of a second augmentedreality navigation system, and/or one or more auxiliary detectorspositioned throughout a surgical environment. In step 1308, a physicalposition and/or orientation indicated by the trajectory selectionguidance augmentation graphic is determined. For example, a surgeon mayposition and orient a head mounted display such that a trajectoryselection guidance augmentation graphic appears to indicate a positionof a patient along a preferred orientation (e.g., the position resideswithin crosshairs on a display screen). The physical position and/ororientation are determined using the relative position(s) and/ororientation(s) determined in step 1308. For example, the coordinates ofthe physical position and/or orientation indicated by the trajectory maybe determined using a unified coordinate system defined during aregistration procedure. In step 1310, a user input signal is received bythe computer subsystem, wherein the user input signal is generated dueto an action (e.g., gesture) by a surgeon using the augmented realitynavigation system and the action is made when the trajectory selectionguidance augmentation graphic is in a desired position and/ororientation. For example, any user input mechanism described hereinabove may be used.

In step 1312, a trajectory is determined based on the physical positionand/or orientation determined in step 1308. Accordingly, in someembodiments, step 1308 is only performed in response to step 1310. Alsoaccordingly, in some embodiments, step 1308 is performed continuouslyover a period of time [e.g., with a certain frequency (e.g., more thanabout once a second)] and the trajectory is determined based on the mostrecent physical position and/or orientation determined when the userinput signal is received. In step 1314, the trajectory is output to arobotic surgical system (e.g., for use in performing a surgicalprocedure). In optional step 1316, trajectory augmentation graphics thatrepresent the trajectory determined in step 1312 are displayed on adisplay screen of the augmented reality navigation system, for example,in accordance with the embodiments discussed above (in reference toexemplary method 1200). The trajectory augmentation graphics may bemodified (e.g., using a registration and/or unified coordinate system)prior to display on a display screen.

Exemplary method 1300 includes optional steps 1318 and 1320. Either orboth of the optional steps may be a part of a method. In step optional1318, a haptic object comprising the trajectory determined in step 1312is defined. The haptic object may be determined by the augmented realitynavigation system (e.g., using relative position(s) and/ororientation(s) of real-world features detected by a detector of thesystem) and provided to a robotic surgical system by the computersubsystem of the augmented reality navigation system. The roboticsurgical system may then be confined in its motion such that at least aportion of a surgical tool connected to or inserted into the roboticsurgical system cannot move outside of the haptic object (e.g., due tohaptic feedback provided to a surgeon operating the robotic surgicalsystem). In optional step 1320, output from the augmented realitynavigation system causes the robotic surgical system to automaticallyalign with the represented trajectory and/or automatically move alongthe trajectory. In some embodiments, representations of multipletrajectories may be displayed simultaneously and a surgeon may selectone of the represented trajectories to have the robotic surgical systemalign to (e.g., using a pointer tool). In some embodiments, whether oneor multiple representations of trajectories are displayed, theaugmentation graphics used may appear overlaid over a patient anatomy togive a surgeon an accurate view of how the trajectory intersects apatient's anatomy. In some embodiments, portions of a surgical tooland/or its trajectories appear on a virtual display screen viewedthrough a display screen of a head mounted display (e.g., in spatialcorrespondence with a model of patient anatomy such that the trajectoryrepresentation, portion of a surgical tool, and model appear to have thesame relative position and orientation as physical reality). Forexample, a virtual display screen may appear as panel A or panel B ofFIG. 15. Overlaid augmentation graphics may appear as the sketch in FIG.16 illustrates.

A model of patient anatomy may be displayed as overlaid augmentationgraphics during exemplary method 1300 in order to assist a surgeon indefining a trajectory. A plurality of desired trajectories may bedefined. A trajectory selection guidance augmentation graphic may alsobe used to select one of a plurality of defined trajectories (e.g., pre-and/or intra-operatively defined trajectories) by orienting a headmounted display such that a trajectory selection guidance augmentationgraphic appears to coincide with one of the plurality of trajectories. Apreviously defined trajectory may also be updated using exemplary method1300 by loading and/or selecting the predefined trajectory and thenfollowing exemplary method 1300 to revise a position and/or orientationof the trajectory.

Referring now to the block flow diagram shown in FIG. 14, exemplarymethod 1400 is a method of using an augmented reality navigation systemin accordance with some embodiments of the present disclosure. Exemplarymethod 1400 is a method whereby an augmentation reality navigationsystem detects a pointer tool to assist in planning and/or updating oneor more trajectories corresponding to a surgical procedure. In someembodiments, an augmented reality navigation system comprises a pointertool. The computational steps of exemplary method 1400 may be performedby a computer subsystem of an augmented reality navigation system.

In step 1402, one or more detectors mounted on a head mounted display ofan augmented reality navigation system generate a detector input signalthat is received by a computer subsystem of the augmented realitynavigation system. The detector input signal represents a field of viewof the one or more detectors that comprises at least a portion of apatient anatomy (e.g., relevant to a surgical procedure to beperformed). In step 1404, a relative position and/or orientation for oneor more real-world features in the detector input signal (i.e., in thefield of view represented by the detector input signal) corresponding toa pointer tool are determined. The relative position and/or orientationmay be determined in real time or may be taken as an absolute positionand/or orientation represented in a unified coordinate system thatresulted from a registration of an augmented reality navigation systemand a surgical environment (e.g., a registration between the augmentedreality navigation system and objects in the surgical environment suchas a patient, a surgical tool, and a robotic surgical system, forexample). The detector input signal may be received by a computersubsystem from one or more detectors mounted to a head mounted displayof the augmented reality navigation system, one or more detectorsmounted to a head mounted display of a second augmented realitynavigation system, and/or one or more auxiliary detectors positionedthroughout a surgical environment. In step 1406, a position and/or anorientation of the pointer tool are determined based on the relativelyposition and/or orientation for the real-world features determined instep 1404. For example, the coordinates of the physical position and/ororientation of the pointer tool may be determined using a unifiedcoordinate system defined during a registration procedure.

In optional step 1408, augmentation graphics corresponding to arepresentation of a portion of the pointer tool hidden from the naturalfield of view of the surgeon are displayed on a display screen of thehead mounted display. The augmentation graphics may appear overlaid overa patient anatomy such that a surgeon can accurately visual the hiddenportion of the pointer tool while positioning and/or orienting it.Augmentation graphics corresponding to a model of a patient anatomy mayadditionally be overlaid over the patient anatomy to provide furthernavigational information. Alternatively or additionally, the pointertool augmentation graphics may appear on a virtual display screenalongside an oriented model of patient anatomy such that the positionand orientation of the pointer tool relative to the model (e.g., medicalimage data) in the virtual display accurately represents the truephysical relationship between the pointer tool and the patient'sanatomy.

In step 1410, a trajectory is determined and/or updated based, at leastin part, on the position and/or orientation of the pointer tooldetermined in step 1406. In some embodiments, a pointer tool may be usedto select a plurality of points in space that are then used tocollectively determine a trajectory (e.g., a linear or non-lineartrajectory). The trajectory may be determined in step 1410 in responseto a user input signal received by the computer subsystem, wherein theuser input signal is generated due to an action (e.g., gesture) by asurgeon using the augmented reality navigation system and the action ismade when the pointer tool is in a desired position and/or orientation.For example, any user input mechanism described herein above may beused. In some embodiments, steps 1402-1410 are repeated one or moretimes in order to define a plurality of trajectories. In step 1412, thetrajectory (or plurality of trajectories) is output to a roboticsurgical system (e.g., for use in performing a surgical procedure). Inoptional step 1414, trajectory augmentation graphics that represent thetrajectory determined in step 1412 are displayed on a display screen ofthe augmented reality navigation system, for example, in accordance withthe embodiments discussed above (in reference to exemplary method 1200).The trajectory augmentation graphics may be modified (e.g., using aregistration and/or unified coordinate system) prior to display on adisplay screen.

Exemplary method 1400 additionally includes optional steps 1416 and1418. Either or both of the optional steps may be a part of a method. Inoptional step 1416, a haptic object comprising the trajectory determinedin step 1410 is defined. The haptic object may be determined by theaugmented reality navigation system (e.g., using relative position(s)and/or orientation(s) of real-world features detected by a detector ofthe system) and provided to a robotic surgical system by the computersubsystem of the augmented reality navigation system. The roboticsurgical system may then be confined in its motion such that at least aportion of a surgical tool connected to or inserted into the roboticsurgical system cannot move outside of the haptic object (e.g., due tohaptic feedback provided to a surgeon operating the robotic surgicalsystem). In optional step 1418, output from the augmented realitynavigation system causes the robotic surgical system to automaticallyalign with the represented trajectory and/or automatically move alongthe trajectory. In some embodiments, representations of multipletrajectories may be displayed simultaneously and a surgeon may selectone of the represented trajectories to have the robotic surgical systemalign to (e.g., using a pointer tool). In some embodiments, whether oneor multiple representations of trajectories are displayed, theaugmentation graphics used may appear overlaid over a patient anatomy togive a surgeon an accurate view of how the trajectory intersects apatient's anatomy. In some embodiments, portions of a surgical tooland/or its trajectories appear on a virtual display screen viewedthrough a display screen of a head mounted display (e.g., in spatialcorrespondence with a model of patient anatomy such that the trajectoryrepresentation, portion of a surgical tool, and model appear to have thesame relative position and orientation as physical reality). Forexample, a virtual display screen may appear as panel A or panel B ofFIG. 15. Overlaid augmentation graphics may appear as the sketch in FIG.16 illustrates.

A single augmented reality navigation system may be configured toperform, inter alia, each of exemplary method 1200, exemplary method1300, and exemplary 1400. A choice of a particular method to define atrajectory or a particular type of augmentation graphic to display maydepend on the surgical procedure being performed. For example, surgeonsmay prefer augmentation graphics and/or trajectories definition methodswhen performing certain minimally invasive surgery (MIS) procedures thatare different than those used when performing an equivalent traditional(i.e., non-MIS) procedure. Likewise, methods and graphics may depend onparticular surgeon performing a procedure or a particular patient. Thus,in certain embodiments, an augmented reality navigation system can storesettings on a per procedure, per patient, and/or per surgeon basis. Insome embodiments, patient health information is displayed on one or morevirtual display screens while navigational information is displayedusing overlaid augmentation graphics and, optionally, on an additionalvirtual display.

The following is a description of an exemplary use of an illustrativeembodiment of an augmented reality navigation system. After theaugmented reality navigation system receives a patient's medical imagedata, a user interface displayed on a display screen of the augmentedreality navigation system enables planning of a surgical procedure byallowing the surgeon to navigate a model derived from the medical imagedata and position virtual representations of surgical implants and/ortrajectories as desired. Positioning may occur using surgeon input(e.g., via a motion sensor or an auxiliary input device). The augmentedreality navigation system is then used to track positions of a roboticarm and one or more real-world features associated with the patient(e.g., as detected by a detector). Throughout, the augmented realitynavigation system synchronizes its coordinate system with that of thepatient anatomy and robotic surgical system (for example, by periodic orcontinuous re-registration). Upon receiving input from the surgeon, forexample, via a motion sensor, the augmented reality navigation systemcauses the end-effector of the robotic arm to be automaticallypositioned in alignment with a planned trajectory, compensating at alltimes for shifts in the position of the patient and allowing treatmentwhile avoiding critical anatomical structures.

In another exemplary use, a model derived from medical image data isregistered in order to plan an implant trajectory. In certainembodiments, a medical image itself (corresponding to the medical imagedata) is used as a model of patient anatomy. A particular registrationmethod may be chosen based on the imaging technique used to generate themedical image data. The imaging may be done pre- or intra-operatively.Once registered, the surgeon will use augmentation graphics displayed ona display screen of a head mounted display to determine necessarytrajectories and locations for surgical implants. Once planning iscomplete, input from the surgeon can cause a robotic arm of a roboticsurgical system to move automatically onto a planned trajectory. Ifmultiple trajectories are planned, the surgeon may move the end effectorclose to the first planned trajectory. Alternatively, the first plannedtrajectory can be selected using user input (e.g., into a motionsensor). The augmented reality navigation system will indicate whichplanned trajectory is in range on the display screen and the system willslowly move the robotic arm onto the selected (e.g., proximal) plannedtrajectory. Augmentation graphics can be used to indicate once theselected planned trajectory has been achieved and motion can be limitedalong that trajectory. The surgeon will then use surgical tool(s)connected to the robotic surgical system to insert the desired surgicalimplant. The navigation camera will track the positions of the tool(s)in real time and display the models on the display screen, for example,appearing overlaid over the patient anatomy.

In certain embodiments, augmented reality navigation systems disclosedherein can be used by medical specialties that can benefit from bothaugmented reality visualization and precision motion tracking (e.g.,trauma, navigated spine, pre-planned cranial). In certain embodiments,augmented reality navigation systems disclosed herein can be used inopen, percutaneous, minimally invasive surgical (MIS) proceduresperformed in the operating room or interventional outpatient procedures,all of which may contain some form of patient imaging. In certainembodiments, augmented reality navigation systems disclosed herein willbe valuable in any such procedure which requires the surgeon(s) to viewremote displays with critical navigational information and to mentallytranslate that information into the surgical space or within thepatient. For example, a surgical procedure in which an augmented realitynavigation system disclosed herein is used may be a spinal procedure, anorthopedic procedure, an orthopedic trauma procedure, a neurosurgicalprocedure, a minimally invasive procedure or any combination thereof. Incertain embodiments, augmented reality navigation systems are used fortraining, for example, in simulations of surgical procedures using acadaver or prosthetic model of patient anatomy.

Exemplary embodiments of systems and methods disclosed herein weredescribed above with reference to computations performed locally by acomputing device. However, computations performed over a network arealso contemplated. FIG. 17 shows an illustrative network environment1700 for use in the methods and systems described herein. In briefoverview, referring now to FIG. 17, a block diagram of an exemplarycloud computing environment 1700 is shown and described. The cloudcomputing environment 1700 may include one or more resource providers1702 a, 1702 b, 1702 c (collectively, 1702). Each resource provider 1702may include computing resources. In some implementations, computingresources may include any hardware and/or software used to process data.For example, computing resources may include hardware and/or softwarecapable of executing algorithms, computer programs, and/or computerapplications. In some implementations, exemplary computing resources mayinclude application servers and/or databases with storage and retrievalcapabilities. Each resource provider 1702 may be connected to any otherresource provider 1702 in the cloud computing environment 1700. In someimplementations, the resource providers 1702 may be connected over acomputer network 1708. Each resource provider 1702 may be connected toone or more computing device 1704 a, 1704 b, 1704 c (collectively,1704), over the computer network 1708.

The cloud computing environment 1700 may include a resource manager1706. The resource manager 1706 may be connected to the resourceproviders 1702 and the computing devices 1704 over the computer network1708. In some implementations, the resource manager 1706 may facilitatethe provision of computing resources by one or more resource providers1702 to one or more computing devices 1704. The resource manager 1706may receive a request for a computing resource from a particularcomputing device 1704. The resource manager 1706 may identify one ormore resource providers 1702 capable of providing the computing resourcerequested by the computing device 1704. The resource manager 1706 mayselect a resource provider 1702 to provide the computing resource. Theresource manager 1706 may facilitate a connection between the resourceprovider 1702 and a particular computing device 1704. In someimplementations, the resource manager 1706 may establish a connectionbetween a particular resource provider 1702 and a particular computingdevice 1704. In some implementations, the resource manager 1706 mayredirect a particular computing device 1704 to a particular resourceprovider 1702 with the requested computing resource.

FIG. 18 shows an example of a computing device 1800 and a mobilecomputing device 1850 that can be used in the methods and systemsdescribed in this disclosure. The computing device 1800 is intended torepresent various forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers. The mobile computing device1850 is intended to represent various forms of mobile devices, such aspersonal digital assistants, cellular telephones, smartphones, and othersimilar computing devices. The components shown here, their connectionsand relationships, and their functions, are meant to be examples only,and are not meant to be limiting.

The computing device 1800 includes a processor 1802, a memory 1804, astorage device 1806, a high-speed interface 1808 connecting to thememory 1804 and multiple high-speed expansion ports 1810, and alow-speed interface 1812 connecting to a low-speed expansion port 1814and the storage device 1806. Each of the processor 1802, the memory1804, the storage device 1806, the high-speed interface 1808, thehigh-speed expansion ports 1810, and the low-speed interface 1812, areinterconnected using various busses, and may be mounted on a commonmotherboard or in other manners as appropriate. The processor 1802 canprocess instructions for execution within the computing device 1800,including instructions stored in the memory 1804 or on the storagedevice 1806 to display graphical information for a GUI on an externalinput/output device, such as a display 1816 coupled to the high-speedinterface 1808. In other implementations, multiple processors and/ormultiple buses may be used, as appropriate, along with multiple memoriesand types of memory. Also, multiple computing devices may be connected,with each device providing portions of the necessary operations (e.g.,as a server bank, a group of blade servers, or a multi-processorsystem). Also, multiple computing devices may be connected, with eachdevice providing portions of the necessary operations (e.g., as a serverbank, a group of blade servers, or a multi-processor system). Thus, asthe term is used herein, where a plurality of functions are described asbeing performed by “a processor”, this encompasses embodiments whereinthe plurality of functions are performed by any number of processors(e.g., one or more processors) of any number of computing devices (e.g.,one or more computing devices). Furthermore, where a function isdescribed as being performed by “a processor”, this encompassesembodiments wherein the function is performed by any number ofprocessors (e.g., one or more processors) of any number of computingdevices (e.g., one or more computing devices) (e.g., in a distributedcomputing system).

The memory 1804 stores information within the computing device 1800. Insome implementations, the memory 1804 is a volatile memory unit orunits. In some implementations, the memory 1804 is a non-volatile memoryunit or units. The memory 1804 may also be another form ofcomputer-readable medium, such as a magnetic or optical disk.

The storage device 1806 is capable of providing mass storage for thecomputing device 1800. In some implementations, the storage device 1806may be or contain a computer-readable medium, such as a floppy diskdevice, a hard disk device, an optical disk device, or a tape device, aflash memory or other similar solid state memory device, or an array ofdevices, including devices in a storage area network or otherconfigurations. Instructions can be stored in an information carrier.The instructions, when executed by one or more processing devices (forexample, processor 1802), perform one or more methods, such as thosedescribed above. The instructions can also be stored by one or morestorage devices such as computer- or machine-readable mediums (forexample, the memory 1804, the storage device 1806, or memory on theprocessor 1802).

The high-speed interface 1808 manages bandwidth-intensive operations forthe computing device 1800, while the low-speed interface 1812 manageslower bandwidth-intensive operations. Such allocation of functions is anexample only. In some implementations, the high-speed interface 1808 iscoupled to the memory 1804, the display 1816 (e.g., through a graphicsprocessor or accelerator), and to the high-speed expansion ports 1810,which may accept various expansion cards (not shown). In theimplementation, the low-speed interface 1812 is coupled to the storagedevice 1806 and the low-speed expansion port 1814. The low-speedexpansion port 1814, which may include various communication ports(e.g., USB, Bluetooth®, Ethernet, wireless Ethernet) may be coupled toone or more input/output devices, such as a keyboard, a pointing device,a scanner, or a networking device such as a switch or router, e.g.,through a network adapter.

The computing device 1800 may be implemented in a number of differentforms, as shown in the figure. For example, it may be implemented as astandard server 1820, or multiple times in a group of such servers. Inaddition, it may be implemented in a personal computer such as a laptopcomputer 1822. It may also be implemented as part of a rack serversystem 1824. Alternatively, components from the computing device 1800may be combined with other components in a mobile device (not shown),such as a mobile computing device 1850. Each of such devices may containone or more of the computing device 1800 and the mobile computing device1850, and an entire system may be made up of multiple computing devicescommunicating with each other.

The mobile computing device 1850 includes a processor 1852, a memory1864, an input/output device such as a display 1854, a communicationinterface 1866, and a transceiver 1868, among other components. Themobile computing device 1850 may also be provided with a storage device,such as a micro-drive or other device, to provide additional storage.Each of the processor 1852, the memory 1864, the display 1854, thecommunication interface 1866, and the transceiver 1868, areinterconnected using various buses, and several of the components may bemounted on a common motherboard or in other manners as appropriate.

The processor 1852 can execute instructions within the mobile computingdevice 1850, including instructions stored in the memory 1864. Theprocessor 1852 may be implemented as a chipset of chips that includeseparate and multiple analog and digital processors. The processor 1852may provide, for example, for coordination of the other components ofthe mobile computing device 1850, such as control of user interfaces,applications run by the mobile computing device 1850, and wirelesscommunication by the mobile computing device 1850.

The processor 1852 may communicate with a user through a controlinterface 1858 and a display interface 1856 coupled to the display 1854.The display 1854 may be, for example, a TFT (Thin-Film-Transistor LiquidCrystal Display) display or an OLED (Organic Light Emitting Diode)display, or other appropriate display technology. The display interface1856 may comprise appropriate circuitry for driving the display 1854 topresent graphical and other information to a user. The control interface1858 may receive commands from a user and convert them for submission tothe processor 1852. In addition, an external interface 1862 may providecommunication with the processor 1852, so as to enable near areacommunication of the mobile computing device 1850 with other devices.The external interface 1862 may provide, for example, for wiredcommunication in some implementations, or for wireless communication inother implementations, and multiple interfaces may also be used.

The memory 1864 stores information within the mobile computing device1850. The memory 1864 can be implemented as one or more of acomputer-readable medium or media, a volatile memory unit or units, or anon-volatile memory unit or units. An expansion memory 1874 may also beprovided and connected to the mobile computing device 1850 through anexpansion interface 1872, which may include, for example, a SIMM (SingleIn Line Memory Module) card interface. The expansion memory 1874 mayprovide extra storage space for the mobile computing device 1850, or mayalso store applications or other information for the mobile computingdevice 1850. Specifically, the expansion memory 1874 may includeinstructions to carry out or supplement the processes described above,and may include secure information also. Thus, for example, theexpansion memory 1874 may be provided as a security module for themobile computing device 1850, and may be programmed with instructionsthat permit secure use of the mobile computing device 1850. In addition,secure applications may be provided via the SIMM cards, along withadditional information, such as placing identifying information on theSIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory(non-volatile random access memory), as discussed below. In someimplementations, instructions are stored in an information carrier and,when executed by one or more processing devices (for example, processor1852), perform one or more methods, such as those described above. Theinstructions can also be stored by one or more storage devices, such asone or more computer- or machine-readable mediums (for example, thememory 1864, the expansion memory 1874, or memory on the processor1852). In some implementations, the instructions can be received in apropagated signal, for example, over the transceiver 1868 or theexternal interface 1862.

The mobile computing device 1850 may communicate wirelessly through thecommunication interface 1866, which may include digital signalprocessing circuitry where necessary. The communication interface 1866may provide for communications under various modes or protocols, such asGSM voice calls (Global System for Mobile communications), SMS (ShortMessage Service), EMS (Enhanced Messaging Service), or MMS messaging(Multimedia Messaging Service), CDMA (code division multiple access),TDMA (time division multiple access), PDC (Personal Digital Cellular),WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS(General Packet Radio Service), among others. Such communication mayoccur, for example, through the transceiver 1868 using aradio-frequency. In addition, short-range communication may occur, suchas using a Bluetooth®, Wi-Fi™, or other such transceiver (not shown). Inaddition, a GPS (Global Positioning System) receiver module 1870 mayprovide additional navigation- and location-related wireless data to themobile computing device 1850, which may be used as appropriate byapplications running on the mobile computing device 1850.

The mobile computing device 1850 may also communicate audibly using anaudio codec 1860, which may receive spoken information from a user andconvert it to usable digital information. The audio codec 1860 maylikewise generate audible sound for a user, such as through a speaker,e.g., in a handset of the mobile computing device 1850. Such sound mayinclude sound from voice telephone calls, may include recorded sound(e.g., voice messages, music files, etc.) and may also include soundgenerated by applications operating on the mobile computing device 1850.

The mobile computing device 1850 may be implemented in a number ofdifferent forms, as shown in the figure. For example, it may beimplemented as a cellular telephone 1880. It may also be implemented aspart of a smart-phone 1882, personal digital assistant, or other similarmobile device.

Various implementations of the systems and techniques described here canbe realized in digital electronic circuitry, integrated circuitry,specially designed ASICs (application specific integrated circuits),computer hardware, firmware, software, and/or combinations thereof.These various implementations can include implementation in one or morecomputer programs that are executable and/or interpretable on aprogrammable system including at least one programmable processor, whichmay be special or general purpose, coupled to receive data andinstructions from, and to transmit data and instructions to, a storagesystem, at least one input device, and at least one output device.

These computer programs (also known as programs, software, softwareapplications or code) include machine instructions for a programmableprocessor, and can be implemented in a high-level procedural and/orobject-oriented programming language, and/or in assembly/machinelanguage. As used herein, the terms machine-readable medium andcomputer-readable medium refer to any computer program product,apparatus and/or device (e.g., magnetic discs, optical disks, memory,Programmable Logic Devices (PLDs)) used to provide machine instructionsand/or data to a programmable processor, including a machine-readablemedium that receives machine instructions as a machine-readable signal.The term machine-readable signal refers to any signal used to providemachine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniquesdescribed here can be implemented on a computer having a display device(e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor)for displaying information to the user and a keyboard and a pointingdevice (e.g., a mouse or a trackball) by which the user can provideinput to the computer. Other kinds of devices can be used to provide forinteraction with a user as well; for example, feedback provided to theuser can be any form of sensory feedback (e.g., visual feedback,auditory feedback, or tactile feedback); and input from the user can bereceived in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in acomputing system that includes a back end component (e.g., as a dataserver), or that includes a middleware component (e.g., an applicationserver), or that includes a front end component (e.g., a client computerhaving a graphical user interface or a Web browser through which a usercan interact with an implementation of the systems and techniquesdescribed here), or any combination of such back end, middleware, orfront end components. The components of the system can be interconnectedby any form or medium of digital data communication (e.g., acommunication network). Examples of communication networks include alocal area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

In some implementations, modules or computational subsystems (e.g. aposition tracking module and user input module) described herein can beseparated, combined or incorporated into single or combined modules.Modules and arrangements thereof depicted in figures are not intended tolimit the systems and methods described herein to the softwarearchitectures shown therein.

Certain embodiments of the present invention were described above. Itis, however, expressly noted that the present invention is not limitedto those embodiments, but rather the intention is that additions andmodifications to what was expressly described herein are also includedwithin the scope of the invention. Moreover, it is to be understood thatthe features of the various embodiments described herein were notmutually exclusive and can exist in various combinations andpermutations, even if such combinations or permutations were not madeexpress herein, without departing from the spirit and scope of theinvention. In fact, variations, modifications, and other implementationsof what was described herein will occur to those of ordinary skill inthe art without departing from the spirit and the scope of theinvention. As such, the invention is not to be defined only by thepreceding illustrative description.

Having described certain implementations of augmented reality navigationsystems for use with a robotic surgical system and methods of their use,it will now become apparent to one of skill in the art that otherimplementations incorporating the concepts of the disclosure may beused. Therefore, the disclosure should not be limited to certainimplementations, but rather should be limited only by the spirit andscope of the following claims.

What is claimed is:
 1. An augmented reality navigation system for usewith a robotic surgical system, the system comprising: a head mounteddisplay comprising an at least partially transparent display screenconfigured to display augmentation graphics which appear to a user to besuperimposed on at least a portion of a natural field of view of theuser; at least one detector for identifying real-world features, the atleast one detector connected to the head mounted display; a surgicaltool having markers and configured to be detected by at the at least onedetector, wherein a representation of at least a portion of the surgicaltool and/or a trajectory of the surgical tool is presented in the headmounted display, wherein a detector input signal from the at least onedetector corresponds to a field of view of the at least one detector andthe field of view comprises at least a portion of anatomy of a patientduring a surgical procedure, wherein the detector input signal includesa relative location and/or orientation for each of one or more of thereal-world features, wherein the surgical tool is inserted into orconnected to the robotic surgical system.
 2. The augmented realitynavigation system of claim 1, wherein a camera system for detectingreal-world features is electrically coupled to the head mounted display.3. The augmented reality navigation system of claim 1, wherein the headmounted display provides a representation of the surgical tool and atrajectory of the surgical tool overlaid on the anatomy of the patient.4. The augmented reality navigation system of claim 1, further includesa motion sensor connected to the head mounted display for outputting amotion signal based on measured motion of the head mounted display. 5.The augmented reality navigation system of claim 1, wherein the roboticsurgical system includes an arm, an end effector coupled to a first endof the arm and a base coupled to the second end of the arm, wherein thearm moves the end effector that is configured to receive the surgicaltool to a trajectory selected by a user.
 6. The augmented realitynavigation system of claim 5, wherein the robotic surgical system andthe augmented reality navigation system includes a haptic feedbacksystem to control the robotic arm within a selected trajectory.
 7. Theaugmented reality navigation system of claim 6, wherein the surgicaltool includes a communicating with the haptic feedback system thatconstrains the surgical tool to the user selected trajectory.
 8. Theaugmented reality navigation system of claim 1, wherein the at least onedetector comprises a detector with at least a minimum field of view of40 degrees.
 9. The augmented reality navigation system of claim 1,wherein the display screen has a resolution of at least 1280×720 pixels.10. The augmented reality navigation system of claim 1, comprising apointer tool for making surgical planning selections, wherein thepointer tool is configured to be detected by the at least one detector.11. The augmented reality navigation system of claim 1, wherein therobotic surgical system registers anatomy of a patient with theaugmented reality navigation system, and an anatomical model of thepatient based on medical image data.
 12. The augmented realitynavigation system of claim 1, wherein the at least one detectorcomprises a video camera and transmits a video signal to the headmounted display to display augmentation graphics which appear to theuser to be superimposed on at least a portion ao natural field of viewof the user.
 13. The augmented reality navigation system of claim 1,wherein the surgical procedure comprises at least one of a spinalsurgical procedure, an orthopedic surgical procedure, an orthopedictrauma surgical procedure, and a neurosurgical procedure.
 14. Anaugmented reality navigation system for use with a robotic surgicalsystem, the system comprising: a head mounted display comprising an atleast partially transparent display screen configured to displayaugmentation graphics which appear to a user to be superimposed on atleast a portion of a natural field of view of the user; at least onedetector for identifying real-world features, the at least one detectorconnected to the head mounted display; and a computer subsystemconfigured to generate and/or access a representation of at least aportion of a surgical tool and/or a trajectory of the surgical toolduring a surgical procedure, modify at least a portion of therepresentation based on a relative position and/or orientation of one ormore real-world features in a detector input signal received from the atleast one detector, and display, on the display screen, surgical toolaugmentation graphics based on the modified representation, wherein thesurgical tool is inserted into or connected to the robotic surgicalsystem.
 15. The augmented reality navigation system of claim 14, whereinthe computer subsystem is configured to render a surgical toolaugmentation graphic for each of a plurality of surgical tooltrajectories, and display, on the display screen, the plurality ofsurgical tool augmentation graphics such that the surgical toolaugmentation graphics appear overlaid on the anatomy of the patient andeach of the trajectory augmentation graphics indicate a physicaltrajectory that could be followed during the surgical procedure.
 16. Theaugmented reality navigation system of claim 14, wherein the computersubsystem is configured to modify an anatomical model of a patient basedon one or more relative location(s) and/or orientation(s) determinedfrom the detected input signal, thereby forming an updated anatomicalmodel, and the computer subsystem is configured to display, on thedisplay screen, anatomical model augmentation graphics corresponding tothe updated anatomical model such that the updated anatomical modelappears overlaid on the anatomy of the patient.
 17. The augmentedreality navigation system of claim 14, comprising: a motion sensorconnected to the head mounted display for outputting a motion signalbased on measured motion of the head mounted display, wherein thecomputer subsystem is configured to update the surgical toolaugmentation graphics based on motion detected by the motion sensor. 18.The augmented reality navigation system of claim 14, wherein thecomputer subsystem is configured to determine a selected trajectorybased at least in part on a user input trajectory selection signal thatselects the selected trajectory from a set of one or more plannedtrajectories, and automatically move a robotic arm and/or end effectorof the robotic surgical system to be aligned with the selectedtrajectory.
 19. The augmented reality navigation system of claim 18,wherein the computer subsystem is configured to automatically move therobotic arm and/or end effector of the robotic surgical system along thetrajectory.
 20. The augmented reality navigation system of claim 18,wherein the computer subsystem is configured to define a haptic feedbacksystem that comprises the trajectory and constrains motion of a roboticarm and/or end effector such that motion of at least a portion of asurgical tool attached to the robotic arm and/or end effector isconstrained to within the haptic feedback system.